The Leaderboard Illusion 2025 04 30

A leaderboard score can reflect which results a provider chooses to show, not only how good its model is. In April 2025, Cohere published The Leaderboard Illusion, a cross-institutional study of Chatbot Arena, where visitors vote between anonymous answers from two models. The authors analyzed 2 million battles and 243 models from 42 providers between January 2024 and April 2025. They report that some providers privately tested many variants and showed only the best, which they say pushes scores upward.

They also report that proprietary providers received 62.8% of Arena's battle data, which they attribute to hidden testing and uneven sampling. In a controlled experiment, raising the share of Arena data in training from 0% to 70% lifted the win rate on ArenaHard from 23.5% to 49.9%. The authors recommend disclosing every submission, capping private variants (for example, three per provider) and sampling by uncertainty. These are the authors' findings about early 2025, not a statement about Arena's current practices.