PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
Source
arxiv.org
Author
Marco Paina, Mark Niklas M\"uller, Niels M\"undler-Sasahara, Martin Vechev
Date
Why it matters
Coding agents asked to find bugs across a repository recover under 21% of what maintainers fixed, and stronger models raise recall but not precision. Their findings are mostly valid yet skew toward different bugs than humans fix.
articleSWEeper-Bench: Can Agents Discover Bugs in Interactive Software?Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan
articleSWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding BenchmarkSoham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang