Towards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical Study
Source
Qingyuan Li, Chuanyi Li, Yaopeng Yang, Ziwen Ge, Jidong Ge, Bin Luo
Author
Qingyuan Li, Chuanyi Li, Yaopeng Yang, Ziwen Ge, Jidong Ge, Bin Luo
Date
Key takeaways · AI-distilled
With GPT-3.5-Turbo, SRepair gained more on HumanEval-Java than on Defects4J, while SRepair and FixAgent without extrinsic information produced negative gains on BugsInPy.
With GPT-5.4-mini the pattern changed: the methods gained more on Defects4J than on HumanEval-Java, and BugsInPy gains were non-negative but limited, so results depend on the model as well as the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
Code transformations reduced enhancement gains on Defects4J, while benchmark-specific fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → increased gains on BugsInPy, pointing to benchmark-related factors behind the differences.
Directly giving GPT-3.5-Turbo the error messages and triggering tests produced more correct repairs on BugsInPy than the evaluated ChatGPT-enhanced repair methods.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
A caution for anyone benchmarking coding agents: enhancement techniques that beat baselines on one popular repair benchmark can underperform on another, so single-benchmark improvement claims should be treated skeptically.
articleRethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencyJunchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, Fabio Santos
articleRefine After Generation: Toward Correct and Concise Patches in LLM-based Program RepairWenqiang Luo, Jacky Keung, Xiaoyu Shi, Yicheng Sun, Boyang Yang, Zhou Yang, Haoye Tian
articleAppEval: A Unified Benchmark for LLM-Based Mobile Application Repair in ArkTS, Swift, and KotlinBang Xie, Hao Liu, Zhenyu Shi, Yonghao Zhang, Senjian Zhang, Zhiyuan Peng, Xin Yin, Chenhao Ying, Yuan Luo, Wei Chen, Haiming Jin, Shaocong Long, Xu Liu, Zhe Peng