Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
- Source
- withspecific.com
- Author
- theanonymousone
- Date
Public coding benchmarks mostly pull tasks from open source repositories, where models have likely seen similar patterns during training and where business logic is thin. Real-SWE is built to avoid both problems. Its ten tasks come from real, licensed production codebases with the kind of business-rule complexity that open source repositories rarely carry, and eight model-plus-harness combinations were run against the same tasks under the same conditions. The results are far lower than the resolution rates public coding leaderboards usually report. Fable 5.1 running on Claude Code led the field at 38.8% resolution, and the weakest pairing managed only 16.2%, a wide spread that suggests harness design matters nearly as much as the underlying model. For teams choosing a coding agent based on public benchmark scores, this is a useful check: performance on unfamiliar, business-rule-heavy code can look very different from performance on the open source tasks most benchmarks are built from, and the model plus harness combination should be evaluated together, not separately.
Context
Specific Labs' Real-SWE benchmark tests coding agents on tasks drawn from licensed private production codebases. The difficulty includes discovering company-specific rules: its billing example requires understanding tax exemptions, account configuration, and interactions with outside services.
The published 's highest resolution rate is 46.25%. Scores represent single-attempt success averaged across eight independent runs per task, and the evaluates models together with their coding tools. Its useful question is whether an can discover and satisfy requirements within an existing business system. The result gives evidence about those tested combinations and tasks, rather than an isolated model's general coding ability.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Checking sign-in…
Loading comments…