• The method captures each validation command, extracts a test-only patch, and replays it against three code states: the original buggy version, the agent's candidate, and the developer's real fix.
• A test that passes on the candidate and also on the untouched buggy code proves nothing about the reported defect. Almost a quarter of baseline runs closed with a patch whose entire positive evidence was that kind of test.
• Handing the agent the buggy-state replay result cut evidence-inadequate closures by 7.8 percentage points and raised bug-discriminating evidence by 7.4, with no measurable cost to repair success.
• The honest caveat: both effects land under the 10-point threshold the authors preregistered as practically meaningful, and about a third of the gain came from the bare reminder rather than the replay content.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

If you're building or relying on autonomous code-repair agents, this shows nearly half of the passing tests agents cite as proof of a fix don't actually discriminate a correct fix from a no-op — and demonstrates a concrete feedback signal (replaying against the buggy version) that can reduce this blind spot.

• articleWhen Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic SystemsWenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
• postLLMs still produce bugs, but those bugs are different than what they used to be.Boris Cherny
• articleEvaluating Agentic Code Repair Capabilities in Distributed SystemsYibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, Seo Jin Park

Checking sign-in…

Loading comments…