Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Source
Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Bingsheng He
Author
Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Bingsheng He
Published
Why it matters
If you're building LLM-as-judge pipelines, this shows that picking the right auditor model per bias type—not just the most 'unbiased' model overall—meaningfully improves judgment accuracy (0.884 vs 0.824/0.805), giving you a concrete lever to reduce sycophancy, bandwagon, authority, and distraction biases in your eval setup.
This paper studies a technique where a second LLM audits a first LLM's reasoning trace before finalizing a judgment, aiming to reduce cognitive biases like bandwagon, authority, sycophancy, and distraction effects in LLM-as-judge setups.
Testing 9 models across 6 families on 4 bias types and 4 datasets, the authors find that a model's standalone bias resistance does not predict its effectiveness as an auditor, and propose a per-bias auditor selection rule that outperforms both fixed-auditor and no-audit baselines (0.884 vs 0.824 and 0.805 accuracy).
Transcript
arXiv:2607.28636v1 Announce Type: new Abstract: LLMs increasingly serve as automated judges, but their judgments remain vulnerable to cognitive biases. Existing mitigations mostly rely on prompt-driven debiasing, which is brittle across bias types, or human evaluation, which does not scale. We study \emph{Chain-of-Models} (CoM), an automated audit pipeline in which a second model inspects the first model's reasoning trace before producing the final judgment. The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we find that auditor identity matters in two ways. First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet is a weak auditor for Qwen2.5-72B's biased traces. Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy. We operationalize these findings with a per-bias auditor selection rule that, given the bias type, scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, the selector reaches the highest accuracy across the four biased slices ($0.884$ vs.\ $0.824$ for the strongest single fixed auditor and $0.805$ for the no-audit baseline). We release data, configurations, and an LLM-agent skill at https://anonymous.4open.science/r/chain-of-models-B585 .
