How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases
Source
youtube.com
Author
AI Engineer
Date
Context
ARIA's proposed route to self-improvement begins with records of its actual work. In the Weights & Biases team's account, a production interaction becomes an offline test: the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → examines what happened, proposes a code repair and compares the revised version with the deployed one.
The team keeps the research and production agent code identical so experiments exercise the same agent users encounter. It evaluates task success alongside relative comparisons between versions, and turns both successful and unsuccessful interactions into additional tests. This creates a concrete feedback process for maintenance: observed behavior supplies the next test, and the test helps assess a proposed repair.
Key takeaways · AI-distilled
In a live demo, W&B's ARIA agent turns a production trace into an offline evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → task, finds the bug (a missing SDK call), writes a fix and benchmarks the new version against the one in production.
Aysola says benchmarks, evals and agent configs all change together, so measurement has to be airtight; his team keeps the production and research agents byte-for-byte identical.
Agent variants are defined in YAML so many can run side by side in an unconstrained sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition →, and each is scored two ways: pass/fail and relative comparison.
The suite runs 886 tasks, including simulated multi-turn users. Every production miss, and every production win, becomes a new task, so the team spends its time improving the system instead of writing benchmarks by hand.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
videoEvery Correction You Give Your Agent Is Data - Brian Douglas, Paper Compute
videoGive Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta
videoYour Agent Hates Your CLI - Ayush Bhardwaj, Allos AI
videoYou Can't Tell What an Agent Skill Will Do by Reading It - Roey Ben Chaim, Zenity
