Intel / article

LLM Judge Validation Under Sparse Overlap: From Inference to Design

Source: arxiv.org
Author: Junxuan Li, Arko Mukherjee, Soumyabrata Pal
Date: Published Sep 29, 2026

Context

Choosing an depends on how much human covers the same items. A study of sparse annotation tested rankings of ten judges across four real evaluation sets. With only 5% of items labeled by more than one person, the chance of picking the wrong best judge among the ten reached about 65%. The authors report that 25% overlap suffices for judges that are not borderline, but judges whose scores sit near the decision threshold remain hard to classify at any reasonable overlap. The paper also examines drawing the shared items proportionally from informative groups, which reduced false rejections at 5% overlap while raising false approvals. Annotation design therefore affects whether a ranking is credible, alongside the agreement score used to calculate it.

Key takeaways · AI-distilled

The paper proves that sparse overlap, too few items labeled by multiple annotators, is the first-order cause of wrong decisions when validating an judge against humans.
At 5% pairwise overlap, wrong-decision rates reach 25%, and the chance of picking the wrong best judge among ten candidates is 65%, per the authors' analysis.
A derived minimum-overlap formula shows an overlap of at least 0.25 suffices for judges that are not borderline, while borderline cases remain fundamentally hard to decide.
Allocation matters too: a zero-cost stratified sampling scheme halves false-rejection rates versus random sampling when the strata are informative. Results were validated on 10 judges across visual, causal-reasoning and summarization tasks.

Terms in this piece · Glossary

LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Read the source arxiv.org