Quantifying Infrastructure Noise in Agentic Coding Evals
Source
Anthropic Engineering
Author
Anthropic Engineering
Date
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
If you cite or design agentic coding benchmarks, this shows that container resource limits and enforcement strictness can swing scores more than the underlying model does — a critical caveat before trusting or building evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → infrastructure.
Key quotes
“Infrastructure configuration can swing agentic coding benchmarks by several percentage points—sometimes more than the leaderboard gap between top models.”
“The runtime is no longer a passive container, but an integral component of the problem-solving process.”
“The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution.”
“This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped.”
“The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.”
