Vibeleaderboard
← All Intel
Intel / article

Self- and Other-Labels Induce Bidirectional Bias in LLM Judges

Source
arxiv.org
Author
Songeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
Date

Context

A language-model judge can change its score simply because an answer is labeled as its own. In a study of ten models, researchers evaluated selections of narrative constraints instead of generated prose, removing model-specific writing style as an explanation for score differences.

With quality matched, self-labels raised scores and other-labels lowered them regardless of who actually produced the selection. In blind , controlling for selection quality and judge severity removed self-preference on three of four dimensions; on originality, the judges rated their own selections lower. The finding isolates authorship labels as a source of bias in this task, giving evaluators a concrete reason to conceal provenance when it is irrelevant to quality.

Terms in this piece · Glossary
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Recommended reads
Comments

Checking sign-in…

Loading comments…