Glossary · evals

SWE-bench

Also: SWE-bench Verified

The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.

Instead of toy puzzles, SWE-bench hands the model an actual repository and an actual bug report; success means producing a patch that passes the project's test suite. That end-to-end realism — navigate unfamiliar code, localize the fault, fix without breaking — made it the coding-agent scoreboard, with "SWE-bench Verified" as the human-vetted subset frontier labs quote.

Scores depend heavily on the harness wrapped around the model, not just the model. Rapid saturation of the benchmark is itself a data point in how fast coding agents are improving.

In the index

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Swe Bench Sonnet
What Skills Does Swe Bench Verified Evaluate
From SWE-bench to ProgramBench: The Next Generation of Coding Evals
[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu