Quantifying Infrastructure Noise in Agentic Coding Evals
Source
Anthropic Engineering
Author
Anthropic Engineering
Published
Why it matters
If you cite or design agentic coding benchmarks, this shows that container resource limits and enforcement strictness can swing scores more than the underlying model does — a critical caveat before trusting or building eval infrastructure.
Anthropic's research post finds that infrastructure configuration—specifically container resource limits and enforcement methods—can swing Terminal-Bench 2.0 scores by up to 6 percentage points, often exceeding the gap between top-ranked models on leaderboards.
The study shows that strict resource enforcement causes spurious failures (transient OOM kills) while looser headroom both fixes reliability issues and lets agents attempt more resource-intensive but valid solution paths, meaning eval scores partly reflect infrastructure choices rather than pure model capability.
