Shows that benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → results measure the whole agent, model plus scaffolding, and details the design that reached 49%. Helps engineers building coding agents tune scaffolds rather than only swap models.
Key takeaways · AI-distilled
SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → grades each attempt against the real unit tests from the pull request that closed the original GitHub issue, so a pass means the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → matched the human author's functionality.
SWE-bench Verified is a 500-task subset that humans reviewed for solvability, because some original tasks need context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → outside the issue, such as the exact error message to return.
Anthropic's scaffold gives the model only a prompt, a Bash tool and an Edit tool for viewing and editing files, and keeps sampling until the model says it is done or hits its 200k-token context.
The design leaves step-to-step control to the model: the prompt suggests an approach but does not hardcode a workflow or force discrete transitions between stages.
Anthropic notes that if token cost is not a concern, explicitly encouraging the model to produce a long response can help on this kind of task.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
