Vibeleaderboard
← All Intel
Intel / article

Teaching Agents to Code Reliably

Source
arxiv.org
Author
Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan, Shweta Garg, Varun Kumar, Murali Krishna Ramanathan
Date

Context

A paper submitted to ICLR 2027 and posted to arXiv on October 2, 2026 targets a weakness in coding agents that try several patches per issue: a test the writes for its own patch accepts many incorrect patches. The authors argue that varied search locations, varied edit strategies and reliable verification can be trained into the model instead of added by scaffolding.

The abstract reports that directing search by execution feedback and scoring each patch against its own reverted tree, a term it does not define, resolves 52.8% of using 48.1% of the steps an eight-sample baseline spends. On 270 held-out issues, supervised raised pass@1, the share solved on the first attempt, from 31.9% to 35.2%. A reinforcement stage that trains the verifier raised it to 43.0% and verifier precision, the share of accepted patches that are correct, from 26.8% to 41.7%.

These are the authors' own reported results, as given in the abstract. Intel covered a related study on whether passing tests really exercise the bug in repair agents.

Key takeaways · AI-distilled
  • The authors name three failure patterns: attempts keep returning to the same code location so extra samples add no coverage, different methods solve complementary issues no single run reaches, and an agent's own tests accept many incorrect patches.
  • On 270 issues held out from training, weighted supervised fine-tuning raised pass@1 from 31.9% to 35.2% and pass@8 from 46.7% to 51.1%.
  • A reinforcement objective that trains the verifier on gold-labeled repairs and incorrect variants lifted pass@1 to 43.0% and pass@8 to 60.7%, raised verifier precision from 26.8% to 41.7%, and more than halved false acceptance.
  • Out of distribution, the authors report better resolution on two of three suites and better verifier precision on all three, with gains holding at 7B, 14B and 30B against published coder baselines.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Recommended reads
Comments

Checking sign-in…

Loading comments…