VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
Source
arxiv.org
Author
Yueshen Li, Kamer Ali Yuksel, Gokhan Tur, Dilek Hakkani-T\"ur, Jo\~ao Pedro Maia
Date
Why it matters
Self-improving agents need a trustworthy fitness signal. This builds a frozen executable evaluator from the task instruction first, then refines it from counterfactual cases, and shows measurable gains on MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → tasks.
Key takeaways · AI-distilled
VeriEvolve builds the evaluator before optimizing the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →: from the task instruction it synthesizes challenge states and an executable checker, freezes that checker as the fitness signal during each evolution round, and revises it between rounds.
Between rounds, the end states of a strong run and a weak run become a pair of counterfactual test cases for refining the evaluator. The authors report this refinement removed all three false positives the weak executor exposed.
On all 30 MCPMark Filesystem tasks, the evaluators agreed on 141/141 generated and 113/122 held-out states, and evaluator-guided evolution raised externally verified success from 6/30 to 18/30 for GLM-5.2 and from 27/30 to 30/30 for Claude.
Studies on PostgreSQL, Playwright, Notion and GitHub show limits from observability, state representation and search or execution capacity. The authors scope the method to tasks whose success is observable and that have an external verifier for calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition →.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
articleEvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading AgentsSehee Kim, Yumin Choi, Minki Kang, Sung Ju Hwang
articleVerifierBench: Benchmarking the Generation of Executable Verifiers for LLM AgentsJianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang
articleSelf-Evolving Embodied Agents via Skill-Harness EvolutionPeidong Wang, Zhiming Ma, Ying Chang, Xufang Luo, Xiaocui Yang, Shi Feng, Yuqing Yang, Dongsheng Li