Evaluates coding agents on investigating a running codebase, not only on fixing issues. It gives a way to compare agents on root-cause analysis and onboarding-style questions.
Key takeaways · AI-distilled
Scale's SWE Atlas runs agents in reproducible sandboxes built from the same production repositories as SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Pro, but scores investigation and maintenance work (Codebase QnA, Test Writing, Refactoring) rather than only whether a change is correct.
Codebase QnA tasks are net-new questions written by professional engineers, such as tracing how kitten @ ls reaches a running kitty instance through Unix sockets, IPC framing and command dispatch across C and Python, then validating by running it.
The primary metric, Task Resolve Rate, counts a task only when every expert rubric item is satisfied. Models run on the SWE-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → scaffold, and some were also tested in their native Claude Code and Codex CLI harnesses.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
