Every eval you keep is a activation steeringChanging a model's behavior by nudging its internal activations directly at runtime — no retraining, no prompt changes.Full definition →. When one fails you tweak the system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition → or a tool description until it passes, so each retained test applies permanent pressure on the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s behavior in that direction.
Keep SDK plumbing tests — prompt passthrough, interrupt config, multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → routing — out of your capability scores. Every model passes them, so they add zero signal while inflating the suite.
Give each eval a docstring naming the capability it measures, and trace every run to a shared project. That makes the suite self-documenting and lets anyone on the team diagnose a failure or argue an eval no longer earns its slot.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
activation steering — Changing a model's behavior by nudging its internal activations directly at runtime — no retraining, no prompt changes.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
Why it matters
A systematic framework for curating eval data and measuring agent behavior, instead of eyeballing whether it "seems better." Read it if you want your agent improvements to be provable rather than vibes.
Key quotes
“More evals ≠ better agents. Instead, build targeted evals that reflect desired behaviors in production.”
“Every eval is a vector that shifts the behavior of your agentic system.”
“We dogfood our agents every day. Every error becomes an opportunity to write an eval and update our agent definition & context engineering practices.”
“Create that taxonomy by looking at what they test, not where they come from.”
