LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent
Source
Han Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
Author
Han Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
Date
Key takeaways · AI-distilled
LoopsBench reframes coding-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → from agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → engineering to LOOP engineering: 112 long-horizon tasks across 8 languages, each a dependency DAG of separately testable development units.
The flow-aware runtime releases tests along the ready frontier and keeps completed nodes as standing regression obligations - so an agent can lose ground it already won.
Even the strongest configuration (Opus-4.7 + Claude Code with outer continuation) resolves only 25% of tasks - sustained multi-unit development remains far from solved.
Agents recorded plans recover only part of the true prerequisite DAG, and regression events stay visible across every loop profile evaluated.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
It gives agentic engineers a hard number for how badly current coding-agent loops degrade over sustained, multi-step work, and an open benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → to test their own harness against. The finding that recorded plans only partially recover the prerequisite DAG is a concrete failure mode to design around.
Key quotes
“Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development.”
“LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.”
“The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.”
articleLoop Engineering: Building Blocks, Adoption, and ImpactJai Lal Lulla, Vahram Nersesyan, Seyedmoein Mohsenimofidi, Christoph Treude, Sebastian Baltes
articleREAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production UsageSmriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra