What Skills Does Swe Bench Verified Evaluate

In a June 2025 report for Epoch AI, Florian Brand and Jean-Stanislas Denain examined what SWE-bench Verified really tests. The benchmark is 500 real bug fixes taken from twelve popular Python repositories. A model gets what a maintainer would see, the codebase from before a merged pull request plus the issue text, and has to write a patch. The patch is graded by running two sets of tests: ones that failed because of the bug and should now pass, and ones that already passed and must keep passing. Every task comes from a fix that maintainers actually merged, and three independent annotators screened the original SWE-bench pool, removing over two-thirds of it. Epoch therefore estimates that only about 5 to 10 percent of the remaining tasks are flawed, a low error rate for a benchmark.

The weakness is in what that screening selects for. Annotators judged 39% of tasks to be trivial changes under 15 minutes for an experienced engineer who knows the codebase, and another 52% to take between 15 minutes and an hour. Only three tasks were estimated at more than four hours. Epoch cites METR's finding that these estimates are too optimistic, though that check covered just six tasks. Amazon's analysis, also cited in the report, classifies 87% of tasks as bug fixes. Django alone supplies nearly half the benchmark, and five repositories make up over 80% of it. Half the issues date from 2020 or earlier, and because all of the code and issues are public, Epoch rates the contamination risk as high: models have likely seen these repositories, and sometimes the issues themselves, during training. In Epoch's view, scores may therefore overstate how a model would do on an unfamiliar, closed-source codebase.

The report's other main point is that the scaffold, meaning the tools and prompting wrapped around the model so it can navigate and edit files, can matter as much as the model itself. Epoch says a good scaffold can add up to 20%. It gives examples: Claude 3.7 rose from 62.3% to 70.2% with a custom scaffold, and DeepSeek R1-0528 scored 33% on Epoch's own tooling against the 57.6% DeepSeek reports with the Agentless scaffold. As a result, a score earned with one scaffold often does not carry over to another. Some scaffolds also generate several attempts and keep the best one, which raises both scores and cost. Epoch suggests watching the single-attempt pass@1 score, which is closer to what a user of a coding tool experiences.

Epoch's bottom line is that SWE-bench Verified is far more realistic than isolated LeetCode-style puzzles, and that saturating it would be strong evidence that models can autonomously fix small to medium issues in Python repositories. It would not show that they can handle system design, new features or vague requirements. The figures above describe the benchmark and leaderboard as of mid-2025. A later Latent Space episode with Mia Glaese and Olivia Watkins of OpenAI's Frontier Evals team is titled The End of SWE-Bench Verified.