Judge panels that share a prompt template or model family repeat each other's blind spots, so majority vote overstates confidence. This method scores judge correlation and reweights accordingly, beating accuracy-weighted voting by 9–14%.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleAgreement Overstates Evidence: Error Dependence in LLM Judge ConsensusElias Hossain, Niloofar Yousefi, Ser-Nam Lim
articleChain-of-Models: Cross-Model Auditing for Bias-Robust LLM JudgesQian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Bingsheng He