Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
Source
Bhanu Prakash Vangala, Tanu Malik
Author
Bhanu Prakash Vangala, Tanu Malik
Date
Key takeaways · AI-distilled
Across agents given identical tasks, the dependency sets they specified agreed as little as 7% of the time, according to the paper's abstract.
Newer agents showed no meaningful improvement at specifying environments, which the authors read as a failure that model scale or recency does not fix on its own.
The largest gap was between the dependencies an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → declared and those actually installed at runtime; the authors attribute it mainly to environment priors absorbed from training data.
The authors argue environment specification is a distinct quality axis that current code-generation benchmarks do not measure, and call for evaluations that test portability alongside functional correctness.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Generated code that runs can still ship wrong dependency declarations. The study shows coding agents systematically misspecify environments, so verify manifests in a clean environment.
articleGrounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test GenerationMichele Tufano, James McClure, Jos\'e Cambronero, Runxiang Cheng, Sherry Y. Shi, Renyao Wei, Dorothy Chen, Franjo Ivan\v{c}i\'c, Livio Dalloro, Pat Rondon
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleEngineering Reliable Coding Agents: Evaluating and Operating the System Around the ModelStephanie Jarmak