Intel / article

Knowing the Form, Not the Function: Automatically Auditing Answer–Authority Decoupling in Legal Benchmarks

Source: arxiv.org
Author: Hsien-Jyh Liao
Date: Published Aug 5, 2026

Context

A correct legal-exam answer can still cite the wrong governing provision. A study of four language models across 238 Taiwan bar-examination items tested answer correctness and references to legal authority separately. In its criminal-law results, 24.0% to 42.4% of valid responses answered correctly but missed the reference authority. The reverse also occurred: some wrong answers cited the expected provision.

The reported finding supports scoring the answer and its legal basis separately. Missing the reference provision does not by itself show that a citation was invented, and the finding concerns the tested rather than legal practice generally.

Key takeaways · AI-distilled

On 238 Taiwan bar-exam items with verified governing provisions, answer correctness and authority dissociate both ways: 24-42% of criminal-law responses were correct but missed the gold statute, while 15-22% were wrong yet cited it.
Models spontaneously emit authority markers under ordinary prompts that never requested citations - which is what makes the mismatch automatically auditable at scale.
Answer-only scoring counts naturally occurring authority misses as full benchmark successes; the authors propose joint answer-authority for statute-grounded legal benchmarks.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Key quotes

“Legal benchmarks typically score final answers even when models also state legal authority.”

“The two dimensions dissociate in both directions.”

“We therefore propose joint answer--authority evaluation for statute-grounded legal benchmarks.”

Read the source arxiv.org