Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

• Error dependence was even stronger among the high-accuracy frontier judges, including judges from different providers, so mixing vendors does not by itself buy independent errors.
• The pattern of shared errors matters, not just the amount: errors shared by most judges and errors concentrated in a smaller group affect consensus differently and favor different voting methods.
• The authors recommend using a small set of trusted examples to estimate judge accuracy and find shared mistakes, then choosing the voting method on those examples before applying it to new data.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

If you use LLM-judge ensembles for evals, this quantifies how much less independent evidence more judges actually add, and shows shared judge errors can silently invert which model looks better.

• articleWhen LLM judges agree, should we believe them?www.amazon.science
• articleInter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextHaoyuan Li, Snigdha Chaturvedi
• articleAgreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical JudgmentsOctavian M. Machidon, Alina L. Machidon, Vojko Strahovnik, Mateja Centa Strahovnik, Jonas Miklav\v{c}i\v{c}, Marko Robnik \v{S}ikonja

Checking sign-in…

Loading comments…