Clip / Developer Tools

Principle 4: Contamination-free by design

From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈10:40

“This way, it's contamination free by design.”

“So, our approach here is that it should always be novel.”

What’s in it

Explains why AI benchmarks need novel, non-public tasks
Argues public GitHub-sourced benchmark tasks risk contamination
Introduces a private-holdout-set principle for benchmark design

Clip transcript

The fourth principle, contamination free by design. We want to do novel tasks only and we want to make sure that we keep private holdout sets. This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos. So, our approach here is that it should always be novel. This way, it's contamination free by design.