Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
Source
Songeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
Author
Songeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
Published
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Anyone running LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → as judge evaluations is exposed to a bias that standard measurements cannot separate from quality.
articleChain-of-Models: Cross-Model Auditing for Bias-Robust LLM JudgesQian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Bingsheng He
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li
articleOptimismBench: Forecasting Bias and the Alignment Effect in Language Model JudgmentSeonglae Cho, Adriano Koshiyama