← All IntelClip / Developer ToolsWeak verifiers cause false positives/negatives
From The Good, the Bad, and the Ugly: Why Coding Benchmarks Are Broken · ≈5:18
“So, this is what a weak verifier looks like.”
“So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it.”
What’s in it
- Exposes flawed test suites baked into SWE-Bench Pro benchmark
- Shows real example of a benchmark test rejecting correct code
- Numbers reveal 8.5% false accepts, 24% false rejects in benchmark
Clip transcript
All right, moving on. The second problem, weak verifiers. Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro. And um let me just fix here so I can see the numbers. In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations. And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers. So, in this example, the test is is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way? So, this test is cornering the LLM and basically uh causing uh those false negatives. In the other example, it's base the test is basically checking functions that are unexported. So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it. So, this is what a weak verifier looks like.
Comments
Sign in to comment.
Loading comments…