SWE-rebench
swe-rebench.com- Category
- Developer Tools
- Rank
- No. 2466Tools index
Previous survey · No. 2253 ·
- Listed in
- Find AI benchmarks
- Pricing
- Free
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Date
About
A continuously refreshed software-engineering benchmark that draws new tasks from recent repository activity, so models are scored on problems published after their training cutoff. The current window covers 111 problems across 65 repositories, reported as resolved rate and pass@5 alongside cost and token use per problem, with rows flagged for potential contamination, external scaffolding, or falling outside the evaluation window. Problems ship as a HuggingFace dataset with a Docker image per problem.
What it does
SWE-rebench's site runs a leaderboard that ranks both bare models and full coding agents, such as Claude Code, Codex and Cursor, on software-engineering problems gathered inside an adjustable date window. Each row reports resolved rate, pass@5, dollar cost and token use per problem, and readers can filter by agent, model or language.
Stated on the product site
- Reported metrics
- Resolved rate, pass@5, cost per problem and tokens per problem
- Entrant types
- Complete agents are ranked alongside standalone models
- Result flags
- Rows can be marked for potential contamination, an external system, or falling beyond the evaluation range
- Problem selection
- Problems are drawn from a selectable time window
- Filters
- Agent, model and language views
- Linked materials
- A paper, a data release and a separate v2 leaderboard
Not stated on the site
- The page text does not explain how problems are collected from repositories or how potential contamination is judged.
- Which agent scaffold runs the plain model rows is not stated in the extracted text.
- Many older models show N/A for the current window, and the page text does not say whether they were not run or simply fall outside it.
- The extracted text does not show a dataset license or a process for submitting a new entry.
Written from the product site at swe-rebench.com.
Tags
Media

Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.