Vibeleaderboard
Index / tool

SWE-rebench

swe-rebench.com
Visit swe-rebench.com
Category
Developer Tools
Rank
No. 2466Tools index

Previous survey · No. 2253 ·

Pricing
Free
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

A continuously refreshed software-engineering benchmark that draws new tasks from recent repository activity, so models are scored on problems published after their training cutoff. The current window covers 111 problems across 65 repositories, reported as resolved rate and pass@5 alongside cost and token use per problem, with rows flagged for potential contamination, external scaffolding, or falling outside the evaluation window. Problems ship as a HuggingFace dataset with a Docker image per problem.

What it does

SWE-rebench's site runs a leaderboard that ranks both bare models and full coding agents, such as Claude Code, Codex and Cursor, on software-engineering problems gathered inside an adjustable date window. Each row reports resolved rate, pass@5, dollar cost and token use per problem, and readers can filter by agent, model or language.

Stated on the product site

Reported metrics
Resolved rate, pass@5, cost per problem and tokens per problem
Entrant types
Complete agents are ranked alongside standalone models
Result flags
Rows can be marked for potential contamination, an external system, or falling beyond the evaluation range
Problem selection
Problems are drawn from a selectable time window
Filters
Agent, model and language views
Linked materials
A paper, a data release and a separate v2 leaderboard

Not stated on the site

  • The page text does not explain how problems are collected from repositories or how potential contamination is judged.
  • Which agent scaffold runs the plain model rows is not stated in the extracted text.
  • Many older models show N/A for the current window, and the page text does not say whether they were not run or simply fall outside it.
  • The extracted text does not show a dataset license or a process for submitting a new entry.

Written from the product site at swe-rebench.com.

Tags

benchmarkswecoding-agentscontaminationleaderboardevaluation

Media

SWE-rebench

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.