Passing benchmark tests isn't the same as being correct.
- Source
- amazon.science
- Author
- Amazon Science
- Date
Amazon Science introduced BenchProofer, which generates formal specifications, reference implementations and Lean proofs for coding-benchmark tasks. Its SWE-Proof set covers the 500 tasks in SWE-Bench Verified. Auditing Claude Opus 4.8's patches that passed the original tests, BenchProofer flagged 27% of them, and human reviewers found that about half of the flagged patches contained real defects. Supplying the generated specifications to the model also raised verified pass rates. SWE-Bench Verified scores count a patch as solved when the repository's tests pass, so some of each reported score rests on patches that pass tests and still carry bugs. BenchProofer derives its specifications from existing test suites, which lets the same method audit agent patches outside the benchmark and build stricter benchmarks.

Context
Coding benchmarks decide whether an fixed a bug by running a handful of tests, and a fix can pass every test while still being wrong on inputs nobody tested. Amazon's open-source BenchProofer tool tries to close that gap with formal verification, and its run over a widely used makes the gap concrete. According to the Amazon Science post, the team ran it over , 500 bug-fix tasks from real open-source projects such as matplotlib, on which frontier models score very highly when judged by tests alone. Its audit flagged nearly a third of the test-passing fixes from Claude Opus 4.8, and human reviewers then found that about half of a sample of those flags were real bugs.
For each task, BenchProofer starts from the accepted reference fix and produces three things. The first is a specification, a precise, machine-checkable statement of what a correct result must satisfy. The second is a version of the fix written in a language a verifier understands, such as Lean. The third is a proof that this version meets the specification, which a proof checker confirms automatically, much as a compiler checks types. Real codebases are too large to verify whole, so the tool proves only the newly written code. Existing functions that the new code calls are covered by contracts, short claims about their behavior that are checked by running them on many generated inputs but are never proven, and the post states that the proofs hold only as long as those assumptions do. In audit mode, the specification guides extra tests that look for inputs where an agent's test-passing patch behaves differently from the reference fix.
Claude Opus 4.8, from Anthropic, passed the original tests on 425 of the 500 tasks. The audit flagged 134 of those 425 patches, nearly one in three, for behaving differently from the reference fix. A difference is not automatically a bug, so human reviewers examined a random 41 of the 134: 21 had real defects, such as missed corner cases, and 20 were valid fixes. A test pass on this benchmark therefore overstates correctness, but an audit flag is a reason for human review, not a verdict on its own. The post gives no human-reviewed sample for OpenAI's GPT-5.5, whose audit flagged 239 of its 406 test-passing patches, so how many of those are real bugs is unknown.
Writing the specification is the hard part. When Opus had to write its own Lean specification from only the bug report and the repository, just 61.4% passed an adversarial review by language-model reviewers, mostly because the specifications left out behavior the fix required. Both models reached more verified passes when given BenchProofer's specification instead of writing their own (Opus 4.8 rose from 84.6% to 93.6%, GPT-5.5 from 79.6% to 94.4%), though the authors note that the supplied specification also points the agent to the relevant functions and that its faithfulness still requires validation. Google DeepMind's AlphaProof Code work on Signal's protocol code kept people on the same step: human experts reviewed every specification before AI agents wrote the Lean proofs. Amazon has released the BenchProofer code and the verified versions of all 500 tasks with the verdict of every check, so others can test whether its specifications hold up.
- BenchProofer turns each task's reference fix into a formal specification, a reference implementation in a verifiable language and a machine-checked proof. It verifies only newly written code and relies on fuzz-tested, unproven contracts for the existing functions that code calls.
- Opus 4.8 passed SWE-Bench Verified tests on 425 of 500 tasks, but spec-guided audit tests flagged 134 of those patches, nearly one in three. Human reviewers checked 41 flagged patches and found 21 real defects; the other 20 were valid fixes.
- Given BenchProofer's specification instead of writing their own, verified pass rates with Lean rose from 84.6% to 93.6% for Opus 4.8 and from 79.6% to 94.4% for GPT-5.5, Amazon reports.
- Writing faithful specs is still hard for agents: only 61.4% of Opus's own Lean specifications passed automated adversarial review, mostly because they left out behavior the bug fix required.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Checking sign-in…
Loading comments…


