Benchmark: AI doesn't find bugs unless you tell it what's wrong
- Source
- swesweep.com
- Author
- lieret
- Date
SWE-sweep is a new benchmark built from 100 repositories in 22 languages, holding about 4,000 real bugs. Unlike issue-resolution benchmarks, it gives the model no ticket. The agent has to discover that something is broken and then fix it. Under that setup the best configuration fixed under 5% of the bugs, and the top model cost thousands of dollars per run. The contrast is the useful part: models are already good at fixing bugs someone has reported, so the weak step is noticing the bug in the first place. For teams planning autonomous code review or background bug-hunting agents, that sets expectations. A sweep that runs unattended over a codebase will be expensive and will miss most defects. Pairing agents with a concrete signal, such as a failing test, a user report or an error trace, is still where they earn their cost.

Context
SWE-sweep tests whether coding agents can find useful repair work without being handed a bug report. Its paper and project page describe 100 repositories with about 4,100 real bugs. An receives a repository and a broad instruction to find and fix bugs, with no issue text, filenames or other hints. A fix counts only if the bug's hidden tests, taken from the matching pull request, pass and the repository's original tests still pass.
The best result on the leaderboard was OpenAI's Sol 5.6 at extra-high reasoning effort, which resolved 4.7% of bugs using the mini-SWE-agent framework. The authors report that Claude Code and Codex did not significantly outperform that framework in an ablation. On this , locating and repairing unspecified defects remained hard for the tested agents; the result does not mean agents find no bugs.
- SWE-Sweep tests whether models find and fix bugs nobody has reported yet. It draws on about 4,000 real GitHub bugs across 100 repos in 22 languages, filtered so each one is solvable in that setting.
- The best setup the authors tested, Sol 5.6 at xhigh effort, scored only 4.7% at a reported cost of about $7,230. Luna 5.6 at xhigh reached 2.5% for about $224.
- Other setups trailed further: Opus 5 at xhigh scored 1.3% for about $5,363, Kimi K3 0.6%, and GPT-5.4 Mini and Gemini 3.5 Flash Lite 0.5% or below. The benchmark is MIT-licensed on GitHub under facebookresearch.
Checking sign-in…
Loading comments…


