← All IntelClip / Developer ToolsPrinciple 4: Contamination-free by design
From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈10:40
“This way, it's contamination free by design.”
“So, our approach here is that it should always be novel.”
What’s in it
- Explains why AI benchmarks need novel, non-public tasks
- Argues public GitHub-sourced benchmark tasks risk contamination
- Introduces a private-holdout-set principle for benchmark design
Clip transcript
The fourth principle, contamination free by design. We want to do novel tasks only and we want to make sure that we keep private holdout sets. This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos. So, our approach here is that it should always be novel. This way, it's contamination free by design.
Comments
Sign in to comment.
Loading comments…