How SWE-Serve Exposes the Gap Between Local Tests and Live Serving
Source
Elizabeth Goodman
Author
Elizabeth Goodman
Date
Key takeaways · AI-distilled
On SWE-Serve's 19 live-serving tasks, the same 627 AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → patches passed 69.4% of the time without live-serving tests but only 45.9% with them: 147 patches that passed every other check failed once a real SGLang server ran.
NVIDIA built SWE-Serve's 53 tasks from 83 merged SGLang pull requests across six families; the median reference fix changes 553 lines across seven files, and patches are judged by hidden verifiers, not diffed against the reference.
Tasks spanning more than one runtime domain (request handling, scheduling, model execution, KV-cache) passed 47.7% versus 69.0% for single-domain tasks, and every model setting showed the same direction.
With mini-swe-agent, Claude Opus 5 and GPT-5.6 Sol both reached about 75% pass@1, but cost varied widely: four models tied at 64% ranged from $0.95 to $7.24 per task. Native harnesses (Codex, Claude Code) did not beat the minimal agent.
Evaluations were closed-book after an open-network pilot showed models fetching upstream code; across 1,749 audited trials, 196 prohibited retrieval attempts were blocked and none succeeded, per the authors.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
SWE-Serve quantifies an underappreciated failure mode: coding agents that pass unit tests still break when a service is actually running, directly relevant to anyone relying on agents to patch production systems.
articleHow to Evaluate AI Agents From Tool Calls to Task Completion
articleTurn Your Latest Observations Into Timely Weather Decisions With NVIDIA Earth-2
articleBenchmarking LLM Inference at Scale with AIPerf
articleTensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
