Hallucination gate reshuffles the Harvey legal agent benchmark
- Source
- x.com
- Date
The updated scoring is strict by design. A task counts only if the agent satisfies every rubric item and its output contains no material hallucination. Under that rule, the best model, Grok 4.7, completes 9.4% of tasks, and more than 60% of outputs that would otherwise have passed contain at least one material hallucination. The result says less about legal agents specifically than about how pass rates are usually reported. A rubric that checks for the presence of correct content can reward an answer that also asserts something false, and in legal work one invented citation can make the whole document unusable. Teams evaluating agents for high-stakes domains should add an explicit check for fabricated claims to their own evals rather than trusting aggregate rubric scores.
Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a hallucination check and require correct responses to not include material misstatements. LAB-AA v1.1's new headline metric, Hallucination-Gated All-Pass Rate, only credits a task when the deliverables satisfy every rubric criterion and contain no material hallucinations. We define a material hallucination as one that would mislead a reader on a substantive point, such as a wrong contractually required date, while a minor hallucination is a real error that is unlikely to meaningfully affect the legal interpretation of a deliverable. Minor hallucinations are reported separately and do not affect the headline score. Grok 4.7 (xhigh) leads at 9.4% Hallucination-Gated All-Pass Rate, and >60% of otherwise passing results across the models tested at launch contain a material hallucination. This is the first step in enhancing the methodology for Harvey LAB-AA. In future updates, we’re working with Harvey to better account for the full set of factors lawyers value, including usability features like style and tone. Key takeaways: ➤…

Context
Artificial Analysis now scores its version of the Legal so that a legal deliverable containing a substantive factual error does not count as a pass. Version 1.1, built with Harvey, makes the -Gated All-Pass Rate its headline metric: a task is credited only when the deliverables meet every rubric criterion and contain no material hallucination. The post defines a material hallucination as one that would mislead a reader on a substantive point, such as a wrong contractually required date. Minor errors that are unlikely to change the legal reading of a deliverable are reported separately and do not affect the score.
The gate reorders the leaderboard. Grok 4.7 (xhigh) from SpaceXAI leads at 9.4%, just ahead of Muse Spark 1.3 (max) from Meta at 8.9% and GPT-6 Astra (max) from OpenAI at 8.6%. Without the gate, Muse Spark would lead clearly with a 26.7% all-pass rate, but two thirds of those passes contained at least one material hallucination. GPT-6 Astra kept almost all of its passes, going from 8.9% to 8.6%, and moved from joint 10th to 3rd. It averaged 0.03 material hallucinations per task, 4 across all 120 tasks, while Gemini 3.8 Flash (high) averaged 13.96. Across the models tested at launch, more than 60% of otherwise passing results contained a material hallucination.
The hallucinations are judged by AI checker models, and the post says that when six checkers were compared on a 20-task subset, every one of them, including Claude Opus 5.5 (high), found zero material hallucinations in GPT-6 Astra's work. Artificial Analysis calls v1.1 a first step and says later versions will try to account for other things lawyers value, such as style and tone, so today's ranking measures correctness against this rubric and error definition rather than overall usefulness to a lawyer. Its full write-up of v1.1 describes how the check is run.
- Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
- Benchmark — A fixed set of tasks with a stated way of scoring them, used to compare models or setups.
- Hallucination — Generated content that is false or unsupported, even when it sounds plausible.
- grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Checking sign-in…
Loading comments…






