SWE-bench Live
swe-bench-live.github.io- Category
- Developer Tools
- Rank
- No. 1819Tools index
Previous survey · No. 1721 ·
- Listed in
- #5 Find AI benchmarks
- Type
- TOOL
- Builder
- microsoft
- GitHub
- 238 stars
- Latest release
- v1.0-multi-language-multi-os-benchmarking
- Date
About
Continuously updated, multi-language software-engineering benchmark built from recent issues to reduce contamination and track current coding agents.
What it does
SWE-bench Live checks whether an agent’s proposed code patch actually fixes a repository task. It applies test and solution patches inside an executable container, rebuilds the project, runs its tests, parses the results, and records whether broken tests became passing without regressions. The repository also includes machinery for collecting, filtering, validating, and publishing custom tasks.
Why it's ranked here
This is a strong choice for serious coding-agent evaluation because it couples problem statements with executable environments and test-based outcomes. It covers Linux and Windows tasks across several programming languages, supports parallel evaluation, and exposes its curation pipeline. The verdict is tempered by substantial infrastructure needs and separate guidance for legacy Python comparisons.
What's good
The evaluator distinguishes tests that must remain passing from tests that must be fixed. Validation reruns post-patch tests three times and treats any observed failure conservatively. Evaluations accept hosted datasets or local records, can select individual tasks, and write per-task logs plus aggregate results.
Tradeoffs
Running it means managing Python dependencies, container images, long command timeouts, and potentially costly parallel jobs. Repository collection requires GitHub tokens and explicit rate-limit handling. Task quality filtering partly depends on a language model instructed to retain uncertain cases. Fair Python-only comparisons still point users toward an older branch and evaluation method.
How to use it well
Use it when developing or comparing autonomous coding agents whose patches must survive real builds and tests across operating systems or languages. Start with selected tasks and one worker, inspect logs, then scale cautiously. It also suits teams creating executable training tasks. It does not provide the coding agent that generates candidate patches.
Technical notes+
evaluation/evaluation.py loads Hugging Face datasets or local JSONL, injects predicted patches, executes them through SetupRuntime, parses test logs, and writes post_patch_log.txt, status.json, report.json, and aggregate results.json; its timeout is 150 minutes and concurrency uses ThreadPoolExecutor. evaluation/validation.py compares pre-patch and post-patch states, repeats the latter three times, and derives PASS_TO_PASS and FAIL_TO_PASS. curation/crawl_repo.py splits GitHub repository searches by star ranges to work around the 1,000-result ceiling, while curation/filter_repo.py asynchronously filters by forks, issue-plus-pull volume, and language share. curation/llm_filter/verify.py rejects task descriptions judged vague, mismatched, or overly revealing, but its prompt explicitly favors retention when uncertain. pyproject.toml uses setuptools, requires Python 3.10 or newer, declares MIT licensing, lists unidiff twice, and declares a dependency named swebench even though the project itself uses that package name.
Observed
- License
- MIT License, declared by the package classifier and license file reference.
- Primary language
- Python.
- Installation
- Editable pip installation from the repository; Python 3.10 or newer is required.
- Interfaces
- Python module command for evaluation plus Fire-based command-line scripts for curation and validation.
- Platform support
- Linux and Windows evaluation environments.
- Dataset surface
- Consumes public Hugging Face datasets or local JSONL task records.
- Execution model
- Applies patches and runs rebuild and test commands inside executable container environments.
Read from README.md, pyproject.toml, curation/crawl_repo.py, curation/filter_repo.py, evaluation/evaluation.py, evaluation/validation.py, curation/llm_filter/verify.py, curation/llm_filter/split_os.py, curation/swe_task_crawling/utils.py, curation/push_dataset/push_windows.py, curation/push_dataset/push_multilang.py, curation/swe_task_crawling/repo_class.py, curation/swe_task_crawling/split_jobs.py, curation/swe_task_crawling/fetch_pulls.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.