Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing
Source
arxiv.org
Author
Jingzhe Liu, Jiliang Tang, Tong Zhao, Neil Shah, Clark Mingxuan Ju
Date
Why it matters
Adding more tasks to automated harness optimization can make the harness worse. The paper traces this to evidence selection and proposes racing candidate harnesses on a small budget.
Key takeaways · AI-distilled
Agentic agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → evolution tunes an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s instructions, tools and control flow from its own task runs. The authors found existing methods get worse as the task pool grows and trace the drop to how evidence is selected for each optimization step.
Successive Scale Racing builds varied evidence combinations based on task quality and coverage, then races the resulting candidate harnesses under a small compute budget in each evolution round.
SSR reports state-of-the-art results in all six configurations across three benchmarks. As the task pool grew from 40 to 200 its pass rate rose from 64.3% to 68.3% while baselines dropped, and racing cut validation executions by 60%.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao
articleWhere Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM AgentsMichael Nguyen, Wei Chen Tan, Nurul Aisyah Hassan, Arvind Raman, Li Hua Lim, Ahmad Faiz Razak
articleHarness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private SuiteMohsen Arjmandi