An AI leaderboard can favor a provider that gets more chances to submit models. The 2025 paper The Leaderboard Illusion examines private testing and selective publication on Chatbot Arena, where people compare model responses.
The authors document privately tested variants and use simulations to study what happens when a provider publishes only its highest-scoring submission. More attempts increase the chance of selecting an unusually favorable result. Their analysis of selective reporting explains why interpreting a leaderboard requires knowing how submissions were selected, as well as how the published scores were calculated.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
