Towards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical Study
Source
arxiv.org
Author
Qingyuan Li, Chuanyi Li, Yaopeng Yang, Ziwen Ge, Jidong Ge, Bin Luo
Date
Context
Adding self-correction or AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → workflows around a code-repair model did not produce consistent gains across the benchmarks in this empirical study. Using OpenAI's GPT-3.5-Turbo, some evaluated methods performed worse on BugsInPy when external repair information was absent. Supplying error messages and tests that triggered the bugs directly produced more correct repairs than the evaluated enhancement methods on that benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →. Results also changed with the underlying model: gains on BugsInPy were non-negative but limited with GPT-5.4-mini. The comparison shows why a repair workflow's value needs to be assessed together with its model, bug collection and available debugging evidence.
Key takeaways · AI-distilled
With GPT-3.5-Turbo, SRepair gained more on HumanEval-Java than on Defects4J, while SRepair and FixAgent without extrinsic information produced negative gains on BugsInPy.
With GPT-5.4-mini the pattern changed: the methods gained more on Defects4J than on HumanEval-Java, and BugsInPy gains were non-negative but limited, so results depend on the model as well as the benchmark.
Code transformations reduced enhancement gains on Defects4J, while benchmark-specific fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → increased gains on BugsInPy, pointing to benchmark-related factors behind the differences.
Directly giving GPT-3.5-Turbo the error messages and triggering tests produced more correct repairs on BugsInPy than the evaluated ChatGPT-enhanced repair methods.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
articleRethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencyJunchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, Fabio Santos
articleRefine After Generation: Toward Correct and Concise Patches in LLM-based Program RepairWenqiang Luo, Jacky Keung, Xiaoyu Shi, Yicheng Sun, Boyang Yang, Zhou Yang, Haoye Tian
articleAppEval: A Unified Benchmark for LLM-Based Mobile Application Repair in ArkTS, Swift, and KotlinBang Xie, Hao Liu, Zhenyu Shi, Yonghao Zhang, Senjian Zhang, Zhiyuan Peng, Xin Yin, Chenhao Ying, Yuan Luo, Wei Chen, Haiming Jin, Shaocong Long, Xu Liu, Zhe Peng