Self-Improving Agentic Systems with Auto-Evals
- Source
- x.com
- Author
- Gauri Gupta
- Date
Context
An automated -improvement loop needs to preserve earlier fixes while testing new ones. Gauri Gupta's auto-harness announcement describes turning recurring failures into evaluation cases, then using those cases to judge changes to the software surrounding an agent.
The repository makes that gate concrete. A proposed change is checked against a regression suite and the full test benchmark; newly solved failures can then join the suite. This creates a growing record of behavior that later changes must preserve under the configured thresholds. It makes automatic experimentation accountable to accumulated tests, whose coverage still determines which regressions can be detected.
- Cluster failures by the fix they would need rather than by symptom. It forces onto the underlying cause and stops the optimizer overfitting to single cases, because a change that clears a whole cluster tends to generalize.
- Verbose traces flood the main fast. Give sub-agents ownership of their own output so the parent only ever sees a summary — that is what makes long-running self-improvement runs survivable.
- The leverage is in the meta-layer, not the loop machinery: the instructions, bias rules, playbooks, tools and skills you wrap around the optimization matter more than the optimizer itself.
- The bottleneck moved past writing code to everything after it — validating behavior, catching regressions, debugging failures, and keeping evaluations honest as the system evolves and user behavior drifts.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
“The bottleneck has moved. It is no longer writing code. It is everything that comes after: validating behavior, catching regressions, debugging failures, and maintaining evaluations and reliability as systems evolve and user behaviors drift.”
“Point it at your agent. Leave it running. Come back to a better agent with evals.”
“Clustering failures is key: Grouping failures by their proposed fix forces you to attack the underlying cause rather than each symptom in isolation.”
“The regression gate is what makes gains compound: Fixed failures become permanent test cases. The system can't backslide past what it's already solved.”
Checking sign-in…
Loading comments…