SWE-bench
swebench.com- Category
- Developer Tools
- Rank
- No. 355Tools index
Previous survey · No. 350 ·
- Listed in
- #1 Find AI benchmarks
- Type
- TOOL
- Builder
- swe-bench
- GitHub
- 5.8k stars
- Date
About
Evaluates whether language-model agents can resolve real GitHub issues in production codebases, with official leaderboards including SWE-bench Verified.
What it does
SWE-bench takes a repository snapshot, an issue description, and a proposed patch. It creates a Docker environment, applies the patch, runs the repository’s tests, and reports whether the change fixed the target problem without breaking expected behavior.
Why it's ranked here
A strong choice for repeatable model evaluation because it combines realistic maintenance tasks with isolated execution and test-based verdicts. Multiple dataset variants support quicker iteration, engineer-verified cases, multilingual work, and visual software tasks. Its demanding infrastructure keeps it better suited to serious experiments than casual checks.
What's good
The evaluation method is concrete and inspectable: patches must apply and pass repository tests. Docker reduces environmental drift, while saved logs can be regraded without rerunning containers. Dataset aliases simplify switching among full, lite, verified, multimodal, and multilingual collections. Local and Modal execution provide two practical operating modes.
Tradeoffs
Local evaluation requires Docker and substantial hardware. The project recommends 120GB of free storage, 16GB of RAM, and eight CPU cores on an x86 machine. ARM support remains experimental. Cached results ignore patch content, so reusing a run identifier can silently return an earlier verdict. Multimodal test evaluation is unavailable through this repository.
How to use it well
Use SWE-bench when comparing code-fixing models or agents under a shared, test-driven protocol. Start with the lite dataset, validate the reference patches, assign a new run identifier whenever predictions change, and inspect per-instance logs. Move heavy runs to Modal when local resources are tight. It evaluates generated patches; it does not replace the model or agent that creates them.
Technical notes+
The package is configured in pyproject.toml with setuptools, Python >=3.10, a swebench console entry mapped to swebench.cli.cli:main, and optional dependency groups for datasets, inference, tests, and documentation. docs/reference/cli.md documents evaluation, report regeneration, image management, dataset building, collection, and version attachment. docs/faq.md states that Docker is mandatory for consistent local evaluation. README.md documents result caching by run_id and instance_id, plus Modal execution. Version guidance is inconsistent: pyproject.toml requires Python >=3.10, README.md shows 3.8+, and docs/installation.md says 3.9+.
Observed
- License
- MIT License
- Language
- Python 3-only package
- Packaging
- Setuptools package with editable pip installation documented
- Interfaces
- Command-line interface plus Python evaluation and inference modules
- Evaluation runtime
- Docker is required for consistent local evaluation environments
- Dataset access
- Supports dataset aliases, Hugging Face identifiers, and local paths
- Cloud execution
- Evaluation can run through Modal
- Platform support
- x86_64 is recommended; arm64 support is experimental
Read from README.md, pyproject.toml, docs/faq.md, docs/index.md, docs/README.md, docs/installation.md, docs/blog/index.md, docs/api/harness.md, docs/reference/cli.md, docs/api/inference.md, docs/api/versioning.md, docs/guides/datasets.md, docs/assets/collection.md, docs/assets/evaluation.md, docs/guides/evaluation.md.
Intel on SWE-bench
- Validation gains that vanished on sealed SWE-bench tasks
- SWE-bench mechanics reveal why models game tests instead of writing good code
- Unrealistic, leaky prompts (SWE-Bench Pro example)
- Contamination in SWE-bench Verified via Opus memorization
- HY 2.0 ships: 406B MoE, 256K context, SWE-bench Verified 53.0
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.