What Makes Software Issue Resolution Tasks Difficult for Agents?
Source
Ebtesam Al-Haque, Brittany Johnson
Author
Ebtesam Al-Haque, Brittany Johnson
Published
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Makes AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → numbers readable by attaching difficulty to the tasks behind them.
articleThe Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent BehaviorXiangzhe Xu, Hamidreza Saghir, Qianhui Wu, Marc-Alexandre C\^ot\'e, Tong Wang, Kiran Lakkaraju, Kexin Pei, Xiangyu Zhang
articleEngineering Reliable Coding Agents: Evaluating and Operating the System Around the ModelStephanie Jarmak
articleEvaluating Agentic Code Repair Capabilities in Distributed SystemsYibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, Seo Jin Park