Hallucination gating collapses legal agent benchmark scores
- Source
- x.com
- Date
The updated scoring is strict by design. A task counts only if the agent satisfies every rubric item and its output contains no material hallucination. Under that rule, the best model, Grok 4.7, completes 9.4% of tasks, and more than 60% of outputs that would otherwise have passed contain at least one material hallucination. The result says less about legal agents specifically than about how pass rates are usually reported. A rubric that checks for the presence of correct content can reward an answer that also asserts something false, and in legal work one invented citation can make the whole document unusable. Teams evaluating agents for high-stakes domains should add an explicit check for fabricated claims to their own evals rather than trusting aggregate rubric scores.
Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a hallucination check and require correct responses to not include material misstatements. LAB-AA v1.1's new headline metric, Hallucination-Gated All-Pass Rate, only credits a task when the deliverables satisfy every rubric criterion and contain no material hallucinations. We define a material hallucination as one that would mislead a reader on a substantive point, such as a wrong contractually required date, while a minor hallucination is a real error that is unlikely to meaningfully affect the legal interpretation of a deliverable. Minor hallucinations are reported separately and do not affect the headline score. Grok 4.7 (xhigh) leads at 9.4% Hallucination-Gated All-Pass Rate, and >60% of otherwise passing results across the models tested at launch contain a material hallucination. This is the first step in enhancing the methodology for Harvey LAB-AA. In future updates, we’re working with Harvey to better account for the full set of factors lawyers value, including usability features like style and tone. Key takeaways: ➤…
Context
ArtificialAnlys, working with Harvey, changed the scoring of its Legal in version 1.1 so a task counts only if the deliverable meets every rubric criterion and has no material : a fabricated or wrong detail that would mislead a reader on a substantive point, such as a wrong contractually required date.
On that Hallucination-Gated All-Pass Rate, Grok 4.7 (xhigh) leads at 9.4%, ahead of Muse Spark 1.3 (max) from Meta at 8.9% and GPT-6 Astra (max) from OpenAI at 8.6%. Without the gate, Muse Spark would lead at 26.7%, but two thirds of those passes contained a material hallucination. Astra kept almost all of its passes (8.9% to 8.6%) and moved from joint 10th to 3rd.
Artificial Analysis reports that more than 60% of otherwise passing results across the models tested at launch contained a material hallucination. It calls this the first step in improving the method, which does not yet capture style or tone, so the rankings reflect this definition of correctness.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
Checking sign-in…
Loading comments…