• Retrying from scratch beats handing the model back its failed program plus test output. Below 7B, blind resampling was the strongest condition outright, and it stayed statistically tied at 7B while burning 2.5-5.5x fewer tokens.
• Showing a model its own failed attempt anchors it: it regenerates a near-identical program in 33-68% of retries, against 2-14% under blind resampling. At 1.5B that self-conditioning costs 6.1 points (p=0.006).
• The information in execution feedback added nothing measurable over a content-free "your code failed" placebo. The usual self-repair comparison confounds the value of the feedback with the value of simply getting another attempt.
• The damage is self-conditioning, not context length. Pasting in solutions to unrelated tasks moved scores by at most plus or minus 3.5 points.
• Asking the model to reflect in words is the only condition that measurably loosens the anchor, but it costs too many tokens to win. Penalty size is predicted by baseline quality alone (r=0.96).

• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

If the retry, not the feedback, is doing the work, self-repair loops are spending tokens for nothing.

• postTraining models to explain their own failures, then rewarding the explanationWRITER
• articleValidation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?Xiaonan Xu, Wenjing Wu
• articleThe Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code ContextAlexander Adkins, Teimuraz Trapaidze

Checking sign-in…

Loading comments…