Intel / article

Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards

Source: arxiv.org
Author: Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)
Date: Published Sep 4, 2026

Context

Inflated scores need not change how models are ordered on a leaderboard. This study compares responses to original test questions with responses to paraphrases intended to preserve the required . It uses deliberately contaminated training runs to calibrate that comparison, then examines 47 public models across four benchmarks. The authors report a 0.997 rank correlation between ordinary and paraphrase-controlled leaderboards, indicating very similar ordering in this sample. Their explanation is that contamination affected models sufficiently similarly to raise scores without substantially reshuffling them. The finding separates two judgments: a leaderboard may retain comparative value even when its raw scores overstate performance on unfamiliar questions. Study abstract.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.

Read the source arxiv.org