Quantifying Infrastructure Noise in Agentic Coding Evals

A coding agent's benchmark score can change substantially without changing the model. Anthropic tested the same model, agent software, and tasks under six resource configurations on Terminal-Bench 2.0. Success differed by six percentage points between strict resource limits and an uncapped setup.

The mechanism had two parts. Moderate extra capacity reduced infrastructure failures without a statistically significant success improvement. More generous allocations then enabled approaches involving larger dependencies and memory-intensive tests, increasing task success. Comparing scores therefore requires knowing both the resources available and how their limits were enforced. Otherwise, an apparent model advantage may partly reflect an easier computing environment.

“Infrastructure configuration can swing agentic coding benchmarks by several percentage points—sometimes more than the leaderboard gap between top models.”

“The runtime is no longer a passive container, but an integral component of the problem-solving process.”

“The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution.”

“This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped.”

“The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.”