One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models
Source
arxiv.org
Author
Siqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, Martin Hirzel
Date
Context
A coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can improve by revising the instructions and tools surrounding its model. In this study, agents inspected previous attempts and edited their prompts, tools and memory across eight programming languages and three base models. The authors report higher solve rates on held-out problems in most combinations, alongside two regions with no improvement. Research abstract.
Some learned procedures transferred across languages, but language-specific engineering details still needed local adaptation. The authors interpret the gains as repairing execution failures: helping an agent use capabilities it already has, with a reusable core and an environment-specific remainder.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleWhere Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM AgentsMichael Nguyen, Wei Chen Tan, Nurul Aisyah Hassan, Arvind Raman, Li Hua Lim, Ahmad Faiz Razak
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao
articleThe Evolution of the Agent HarnessDan McAteer