← All IntelClip / Developer ToolsPrinciple 5: Leaderboards need to explain 'why', not just 'who wins'
From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈11:14
“Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why.”
“And so, we want to basically put the x-axis back on um on on the first page.”
“And people have to dig uh a lot and do their own experiments to get to those data points.”
What’s in it
- Argues benchmark leaderboards should explain 'why', not just rank winners
- Proposes putting key data axes back on the leaderboard's front page
- Critiques current benchmarks for burying useful data behind extra digging
Clip transcript
design. And the fifth and last principle here is information about leaderboards. The benchmark needs to tell a story and needs to help people make decisions. Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why. And so, we want to basically put the x-axis back on um on on the first page. Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront. And people have to dig uh a lot and do their own experiments to get to those data points.
Comments
Checking sign-in…
Loading comments…