Continue or Restart? The Limits of Early Failure Prediction in Coding Agents
Source
arxiv.org
Author
Srinath Srinivasan, Lunxiao Li, Tim Menzies, Wesley Assun\c{c}\~ao
Date
Why it matters
Killing a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → run early to save compute sounds appealing, but early trajectory signals do not separate good from bad attempts on the same task. Restart-or-continue heuristics based on early prefixes are unreliable.
Key takeaways · AI-distilled
Early prefixes modestly help rank which tasks are likely to fail, but not which attempt on the same task will fail, the authors report. A three-seed neural monitor over 32,406 attempts at 20k tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → scored a mean within-task AUC of 0.50 (interval 0.48 to 0.51).
Under a 75k-token cap, learned restarting beat matched random restarting by up to 1.98 points. The authors' algebraic decomposition attributes almost all of that to not stopping runs on easy issues; picking better attempts within a task added -0.11 to +0.24 points.
Simply letting runs continue beat both restart policies on solve rate at lower mean token cost in all four model-framework settings at the 75k cap, and in 216 of 224 budget-threshold configurations tested, according to the abstract.
The authors argue that any early-stopping evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → should show attempt-level discrimination within a task and net compute savings against uninterrupted continuation.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleKnowing When to Stop: The Evidence-Action Gap in Coding AgentsTianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou, Tongshuang Wu
articleCompact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding AgentsRuopei Sun, Jianfeng Cai, Kangwen Zhao, Wengang Zhou, Houqiang Li
articleFinishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating ChallengeYuanchen Bai, Zijian Ding, Angelique Taylor