Coding-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → benchmarks with fixed instructions can miss the rework that follows when users clarify requirements after implementation begins. Researchers examined 3,553 eligible conversation sessions and, where repository history could be replayed, measured deletion or replacement of earlier agent-written lines. Newly stated requirements were followed by roughly twice as much of this code invalidation as matched edits without new requirements. The authors explicitly treat that relationship as observational, not proof of causation. The result makes changing requirements a measurable part of agent evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →: success includes responding to what users discover they need after seeing the first implementation. Study abstract.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleReality Is the Final Verifier: On Two Key Gaps in Agentic Software EngineeringAlexander Krentsel, Shubham Agarwal, Mert Cemri, Shu Liu, Sidharth Sankhe, Ziming Mao, Matei Zaharia, Ion Stoica
articleA Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent AdoptionYegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo
articleFrom Code to Requirements: Agentic Reverse Engineering of Business Rules at Enterprise ScaleGarima Agrawal, Prasun Das, Priyanka L, Minisha N, Akshay Sarvade, Hemath Manivanan, Sravani Joshna, Prasad Kalyansundaram