Intel / article

How SWE-Serve Exposes the Gap Between Local Tests and Live Serving

Source: developer.nvidia.com
Author: Elizabeth Goodman
Date: Published Sep 23, 2026

From the Daily Brief

SWE-Serve is built from 53 real engineering tasks in SGLang, the open inference engine, and grades coding agents on whether their patches work when the service is actually running. That second check is the point. NVIDIA reports that patches passing the conventional test suite still fail live-serving checks roughly a third of the time. Across the 11 models tested, pass@1 ranges from 35% to 75.5%, so even the strongest result leaves a quarter of tasks unsolved. The gap matters most for teams that let agents patch production systems, where a green unit test run is often treated as the merge signal. Serving code fails in ways unit tests rarely exercise: concurrency, memory pressure, request batching and long-running state. The practical lesson is to add a live or staged serving check to any agent pipeline that touches infrastructure, rather than trusting the test suite alone.

Read the 2026-09-24 Brief →

Context

Testing a coding 's patch through a running model server catches failures that other checks miss. In NVIDIA's SWE-Serve evaluation, the same patches passed 69.4% of the time when live-serving tests were excluded, compared with 45.9% under the complete verifier. The comparison covers 19 tasks involving SGLang, software that serves language models.

Those additional tests load models and send requests through the serving interface, exercising components together. About one-third of patches that passed the remaining checks failed this fuller test. The result supports checking the actual request-to-response path before accepting an agent's work; the itself does not establish production readiness.

Key takeaways · AI-distilled

On SWE-Serve's 19 live-serving tasks, the same 627 agent patches passed 69.4% of the time without live-serving tests but only 45.9% with them: 147 patches that passed every other check failed once a real SGLang server ran.
NVIDIA built SWE-Serve's 53 tasks from 83 merged SGLang pull requests across six families; the median reference fix changes 553 lines across seven files, and patches are judged by hidden verifiers, not diffed against the reference.
Tasks spanning more than one runtime domain (request handling, scheduling, model execution, KV-cache) passed 47.7% versus 69.0% for single-domain tasks, and every model setting showed the same direction.
With mini-swe-agent, Claude Opus 5 and GPT-5.6 Sol both reached about 75% pass@1, but cost varied widely: four models tied at 64% ranged from $0.95 to $7.24 per task. Native harnesses (Codex, Claude Code) did not beat the minimal agent.
Evaluations were closed-book after an open-network pilot showed models fetching upstream code; across 1,749 audited trials, 196 prohibited retrieval attempts were blocked and none succeeded, per the authors.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Read the source developer.nvidia.com