Agentless

Why it's ranked here

A published result backs the design. The README reports 82 SWE-bench Lite issues fixed (27.3%) at an average of 34 cents each on release, then 40.7% on Lite and 50.8% on Verified once paired with Claude 3.5 Sonnet. The argument behind those numbers matters more than the numbers: a scripted pipeline with heavy sampling and test-based voting outscored the open-source agent loops it was compared against, while staying cheaper and easier to debug. The code is MIT licensed and the complete benchmark runs are downloadable, so the claims can be checked rather than taken on faith.

What's good

Every stage leaves a record. Each phase writes its results and the full model exchange, token counts included, to line-per-record JSON files, so a bad fix can be traced to the stage that went wrong, and a rerun skips issues already finished. File search combines two signals: the model's own pick of up to five files and an embedding search over the folders the model did not rule out. Candidate fixes are normalized before the vote so equivalent changes count together, and a bundled script prices a run from the logged token counts.

Tradeoffs

It is built around the benchmark, not your repository. Every script loads issues from the SWE-bench dataset by ID and checks out the benchmark's own snapshot of the code, so pointing it at a live bug means writing that glue yourself. Search looks only at Python files and skips test files. The dependency list carries no version numbers and pulls the benchmark harness straight from its git repository. One repair step turns the model's written file name into a string by running it as Python code, which executes model output directly. The cost script hard-codes a single per-token price.

How to use it well

Best fit: researchers and teams building their own issue-fixing systems who want a strong, inspectable baseline to beat or borrow from. Run the stages one command at a time as the SWE-bench guide lays out, start with a single issue ID to see what each stage produces, and set the thread count to fit your API rate limits. The expensive knobs are the sample counts: four location sets times ten repair samples yields forty candidate patches per bug before any test runs. Nothing in the pipeline opens a pull request; its final output is a predictions file for the benchmark's evaluator.

Observed

License

MIT

Language

Python, with a Python 3.11 conda environment in the setup instructions

Install surface

Clone, install requirements.txt, export PYTHONPATH; no package install step documented

Dependencies

Nine entries in requirements.txt with no version pins; SWE-bench installed from its git repository

Interface

Command-line Python scripts, one per pipeline stage

Model backends

OpenAI, Anthropic and DeepSeek decoder classes

Target benchmark

SWE-bench Lite and SWE-bench Verified datasets, selected by a dataset flag

Localization scope

Non-Python files and test files are filtered out before file search