The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating
Source
arxiv.org
Author
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
Date
Context
Reading an LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition →'s preference before it generates an answer can exaggerate how strongly it favors the first response shown. A study of this shortcut examines evaluations that inspect the model's initial scores for verdict labels instead of letting it finish its judgment.
The authors report that some judges would not actually begin with a verdict. Forcing a choice at that point often selects the first-presented response. Across the tested conditions, this inflated measured position bias, even though overall judging accuracy changed little in most conditions. Audits therefore need to distinguish the model's opening label scores from its eventual decision.
Key takeaways · AI-distilled
Three Qwen3 judges did not open with a verdict tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → on 12% to 49% of pairs, versus under 3% for Llama-3.1-8B and Phi-3.5-mini. Forcing a first-token read on those pairs returns whichever response was shown first.
On the 924 pairs where a judge did not commit, the forced read flipped on 89.7% when responses were swapped, against 47.5% when the verdict was read after generation.
A smaller failure remains even when a judge leads with a verdict: it sometimes opens with one letter and reasons its way to the other, on 0 to 5.5% of pairs, at a rate uncorrelated with compliance.
The authors recommend reporting how often a judge leads with a verdict token, which costs one forward pass and no labels, next to any position-bias figure, and treating forced-read bias numbers as upper bounds.
Terms in this piece · Glossary
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.