Where Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM Agents
Source
arxiv.org
Author
Michael Nguyen, Wei Chen Tan, Nurul Aisyah Hassan, Arvind Raman, Li Hua Lim, Ahmad Faiz Razak
Date
Context
Splitting an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s improvement budget evenly can leave every part with too little testing to improve. HARNESSEVO separates the instructions surrounding a fixed LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → into roles, task strategy, tool rules, and reflection or control. On the ALFWorld benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →, the authors found useful gains concentrated in the control instructions. Dividing 64 trial runs across all four parts produced no updates, while concentrating effort on control recovered gains with a smaller total budget. The pattern did not carry over to WebShop, where none of the approaches improved results. These experiments support identifying which instructions help before dividing an optimization budget among them. Study abstract.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleOne Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and ModelsSiqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, Martin Hirzel
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao
articleHarness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private SuiteMohsen Arjmandi