← All IntelClip / Developer ToolsVerifier design to reduce false positives/negatives via observable-behavior testing
From DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve · ≈11:46
“We want to ensure that any correct implementation uh anything that correctly solves the problem is rewarded and this will prevent against uh false negatives.”
“We also make sure that um there's the absence of these PR derived tests that rely on naming relying on specific implementations.”
What’s in it
- Explains why benchmark verifiers should judge behavior, not naming
- Shows how SweetBench Pro cuts false negatives and false positives
- Details verifier design tradeoffs for grading LLM-generated code
Clip transcript
more output tokens being emitted. Um and um finally we have uh we verifier design is of course one of the most um important and tricky parts of building good environments. In Sweepbench Pro we have these verifiers that are testing again for specific implementations. It will fail the model if it uh produces a function that may address the objective but is not named or is not defined within a specific module or if there is the absence of specific helpers or other private functions. Um because again these are derived from the solutions that were merged in the actual PR. Um so for us we want to uh emphasize on the observable behavior as much as possible. We want to ensure that any correct implementation uh anything that correctly solves the problem is rewarded and this will prevent against uh false negatives. We also make sure that um there's the absence of these PR derived tests that rely on naming relying on specific implementations. Um and so this will prevent again again uh again uh towards false negatives as well. And we observed through a combination of these um you know considerations we're able to drastically reduce the false negative as well as the false positive rates when we analyzed um our rollouts compared to SweetBench pros uh using both human experts as well as LLM as judge and yes the coverage for us spans across
Comments
Sign in to comment.
Loading comments…