LLM Judge Validation Under Sparse Overlap: From Inference to Design

• The paper proves that sparse overlap, too few items labeled by multiple annotators, is the first-order cause of wrong decisions when validating an LLM judge against humans.
• At 5% pairwise overlap, wrong-decision rates reach 25%, and the chance of picking the wrong best judge among ten candidates is 65%, per the authors' analysis.
• A derived minimum-overlap formula shows an overlap of at least 0.25 suffices for judges that are not borderline, while borderline cases remain fundamentally hard to decide.
• Allocation matters too: a zero-cost stratified sampling scheme halves false-rejection rates versus random sampling when the strata are informative. Results were validated on 10 judges across visual, causal-reasoning and summarization tasks.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

If you validate an LLM judge against sparse human labels, low annotator overlap can pick the wrong judge 65% of the time. The paper gives an overlap threshold (0.25) and a stratified sampling scheme to reduce false rejections.

• articleAuditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL PipelineHaowei Liu, Hsin-Tai Wu, Yi Fang
• articleAgreement Overstates Evidence: Error Dependence in LLM Judge ConsensusElias Hossain, Niloofar Yousefi, Ser-Nam Lim
• articleInter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextHaoyuan Li, Snigdha Chaturvedi

Checking sign-in…

Loading comments…