The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken

A coding benchmark should distinguish a working solution from one that merely matches the test author's preferred implementation. This talk on benchmark design argues for instructions that specify desired behavior and genuine constraints, paired with checks that assess those requirements.

The proposed approach allows implementation flexibility while retaining precise checks where security or business rules demand them. It also calls for publishing information about individual runs, so developers can understand why a model scored well. This is a framework for designing more informative evaluations, rather than evidence that every existing coding benchmark fails.

“I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards.”

Ali

“Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why.”

Ali

“So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders”

Ali

“Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet.”

Ali

“Um benchmarks are not hard. We need to look under the hood.”

Ali