When auto-improving an agent, expect gains from new tools and workflow changes that survive model swaps, not prompt tweaks or reasoning limits. Versioning, capped attempts and held-out scoring make such gains verifiable.
Key takeaways · AI-distilled
Scale's VeRO lets an optimizer AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → edit any part of a target agent. Across 105 runs, with Claude Sonnet 4.5, Claude Opus 4.5 and GPT-5.2-Codex optimizing GPT-4.1 mini targets, individual gains reached as high as 19 points on GAIA.
Optimizers improved tool useA model's ability to call external functions — run code, search the web, edit files — instead of only generating text.Full definition →, file handling, search and workflow logic, but no tweaking helped where the limit was the underlying model's reasoning ability.
VeRO guards against illusory gains: every edit is saved as a separate version, the optimizer gets a fixed number of attempts, and it never sees the questions used for the final score.
When the best optimized agents were rerun on models of other sizes, families and vendors, new tools and workflow changes held up, while prompt edits were uneven and sometimes scored below the unoptimized baseline.
Optimizers still reached for the prompt first: more than half of all changes were prompt edits, the kind Scale found least likely to survive a model swap.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
