Intel / article

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Source: arxiv.org
Author: Elias Hossain, Niloofar Yousefi, Ser-Nam Lim
Date: Published Sep 22, 2026

From the Daily Brief

Evaluation pipelines often add more LLM judges on the assumption that agreement across a panel is strong evidence. The authors test that assumption directly by measuring how often judges make the same mistakes. Average pairwise error correlation across ten judges came out at 0.21, which they translate into an effective sample of about 3.5 independent judges. The consequence is not only wider error bars. Because judges fail together, correlated errors inverted which model appeared better in up to 28% of comparisons. For teams running judge ensembles, the practical step is to measure error dependence on a labeled sample before trusting consensus, and to count agreement among similar judges as weaker evidence than the panel size suggests.

Read the 2026-09-23 Brief →

Context

Agreement between AI judges supplies less evidence when those judges tend to make the same mistakes. In this study, the authors estimate that their main panel of ten judges provided roughly the statistical information of 3.5 independent judges because their errors were correlated. Accounting for shared errors also removed apparent statistical significance from some comparisons between systems. The authors propose using trusted examples to estimate accuracy and discover which judges fail together before choosing a voting method. That changes what a panel's size means: confidence depends on the independence and pattern of its judgments, not simply the number of votes.

Key takeaways · AI-distilled

Error dependence was even stronger among the high-accuracy frontier judges, including judges from different providers, so mixing vendors does not by itself buy independent errors.
The pattern of shared errors matters, not just the amount: errors shared by most judges and errors concentrated in a smaller group affect consensus differently and favor different voting methods.
The authors recommend using a small set of trusted examples to estimate judge accuracy and find shared mistakes, then choosing the voting method on those examples before applying it to new data.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Read the source arxiv.org