Clip / Education

Undisclosed model testing distorts leaderboard results

From Benchmaxxing: The Gap Between Benchmark Scores and Reality · ≈12:47

Concrete documented case of a lab gaming a public leaderboard through non-transparent testing practices.

What’s in it

Concrete documented case of a lab gaming a public leaderboard through non-transparent testing practices.

Clip transcript

interpret because they don't have that contextualizing information. This was a paper um again about Elmarina and talking about how some of the dynamics of how it's run lead to models overfitting on Elmarina. Um in this instance, the specific chart we're seeing is that Meta tested 27 models without disclosing that it was doing so. Um which you know distorts the results.