GLM Legal: cheap open-weight legal model, with a research regression
- Source
- x.com
- Date
Zai just debuted a legal specific model: GLM Legal, and it is the new #1 open-weight model on Harvey’s Legal Agent Benchmark and LegalBench.
Harvey's Legal Agent Benchmark tests full legal assignments using documents, spreadsheets, and presentations. On this benchmark, it scored 19.2%, about 3x higher than the regular GLM 5.3 Flash (6.7%) and more than double the full-size GLM 5.3 (8.3%). GLM Legal met 92.5% of the individual criteria, up from 82.2% for GLM 5.3 Flash. It scored higher because it missed fewer requirements and fully finished so many more tasks. It scores #6 overall and at about $0.15 per task, it costs roughly 70 times less than Gemini 4 Argon (#5), which costs about $10.55 per task.
LegalBench tests short non-agentic legal reasoning questions. GLM Legal improved 2.3pp from GLM 5.3 Flash and the biggest gain came in interpretation tasks, which rose 5pts. It is now the top open-weight model on LegalBench, just ahead of Kimi K3. It also costs half as much per question as GLM 5.3 Flash. Per-token pricing is identical, so the savings come from the legal version using fewer tokens to answer.
Despite the increases on HLAB and Legal Bench, we saw a decrease on LegalResearchBench, which tests legal research using case-law search and the web. Although it scored 4 points behind GLM 5.3 Flash, its answers were mostly correct, earning 86.1% of rubric points. However, it missed a required point more often. It also searched case law much less frequently, about 6 times per question, compared with 16 for GLM 5.3 Flash.
A legal-tuned model scores 19.2% on Harvey's agent benchmark at about $0.15 per task, versus $10.55 for the #5 model. It regresses on case-law research, searching far less, so test it on your own workload before relying on it.
- On Harvey's Legal , GLM Legal met 92.5% of individual criteria versus 82.2% for GLM 5.3 Flash; Vals AI attributes the gain to fewer missed requirements and many more fully finished tasks.
- The 19.2% agent score is more than double the full-size GLM 5.3 (8.3%), not just the Flash base (6.7%), according to Vals AI.
- On LegalBench it gained 2.3 points over GLM 5.3 Flash, mostly from interpretation tasks (up 5 points), and costs half as much per question despite identical per-token pricing because it uses fewer tokens.
- On LegalResearchBench it scored 4 points below GLM 5.3 Flash yet still earned 86.1% of rubric points; it ran about 6 case-law searches per question against 16 for the base model and missed required points more often.
- The model has a 1M-token and 128k max output. Vals AI tested it at default provider settings: temperature 1, Top-P 0.95 and max reasoning effort.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Checking sign-in…
Loading comments…