Shows an agentic loopThe cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.Full definition → (literature search, method proposal, training, testing) closing alignment gaps on held-out evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → and larger models, a concrete data point on automating safety research.
Key takeaways · AI-distilled
Claude worked one alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → failure at a time in a loop of literature search, method proposal, training and testing, scored by the share of the safety gap closed across three to five benchmarks per category.
Methods that hurt general capability were excluded and Claude could not distill its own alignment into the target; a monitoring AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → read every planned method before it ran to enforce those rules.
For all 10 failure categories, fixes improved target benchmarks without capability loss, held up on withheld benchmarks and Petri, and still worked on models up to 4.7 times larger than the ones optimized.
On deception in Gemma-2-2B, Claude averaged 85% of the gap closed across runs versus 20% for six experienced safety researchers under the same rules. Anthropic notes the humans could not iterate, so it is not a direct comparison.
Claude Sonnet 5 aligned an early Opus 4.8 checkpoint to near production scores in 60 hours, using just over 2,000 training examples, which Anthropic estimates is roughly 15,000 times more efficient than its production procedure.
Terms in this piece · Glossary
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
