Instead of toy puzzles, SWE-bench hands the model an actual repository and an actual bug report; success means producing a patch that passes the project's test suite. That end-to-end realism — navigate unfamiliar code, localize the fault, fix without breaking — made it the coding-agent scoreboard, with "SWE-bench Verified" as the human-vetted subset frontier labs quote.

Scores depend heavily on the harness wrapped around the model, not just the model. Rapid saturation of the benchmark is itself a data point in how fast coding agents are improving.