Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
- Source
- arxiv.org
- Author
- Songeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
- Date

Context
A language-model judge can change its score simply because an answer is labeled as its own. In a study of ten models, researchers evaluated selections of narrative constraints instead of generated prose, removing model-specific writing style as an explanation for score differences.
With quality matched, self-labels raised scores and other-labels lowered them regardless of who actually produced the selection. In blind , controlling for selection quality and judge severity removed self-preference on three of four dimensions; on originality, the judges rated their own selections lower. The finding isolates authorship labels as a source of bias in this task, giving evaluators a concrete reason to conceal provenance when it is irrelevant to quality.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleThe First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without GeneratingGnaneswar Villuri, Hashmath Shaik, Alex Doboli
articleRating the Raters: Rasch Measurement Theory for LLM EvaluationPratik S. Sachdeva, Nathan Boudol
Checking sign-in…
Loading comments…