What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs
Source
arxiv.org
Author
Wenbo Ye, Yongxi Zhou, Junwei Yao, Jiahui Ge
Date
Why it matters
When retrying a failed repair AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →, passing the raw transcript did as well as an extra model call to write a lesson, so you can skip the summarization step. The gains were not statistically significant.
Key takeaways · AI-distilled
The study retried 181 failed gpt-5.4-mini attempts on 75 BugsInPy bugs under five conditions with a predeclared analysis. On the bug-revealing tests, every memory condition beat a plain retry's 29.8% by 5.5 to 9.9 points in point estimate, but none significantly.
A post hoc check against each full test suite found that retries which had also seen the previous attempt's test results more often failed other tests the reference fix passes, which reduces their apparent gains.
Outcomes varied far more between individual failures than between memory conditions, and the authors give the sample sizes a confirmatory study would need.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleWhen Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model AgentsWenhui Chu (University at Albany, State University of New York)
articleTowards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical StudyQingyuan Li, Chuanyi Li, Yaopeng Yang, Ziwen Ge, Jidong Ge, Bin Luo
articleAdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program RepairZ. C. Luo, J. C. Guo, W. J. He, S. Y. Wang, J. C. Yu, F. M. Zhao, Y. Chen, T. Cao, L. Q. Liu, N. Zheng, W. Xu, J. Jiang, Z. M. Zhao