Anthropic put Claude in charge of aligning other models for 48 hours
- Source
- x.com
- Date
New Fellows Research: Can Claude autonomously align other AIs? We gave Claude 48 hours and 1 GPU to improve the alignment of small models. It researched and proposed methods, then trained and tested the models on its own. It worked surprisingly well. https://t.co/nhlCMgQl46
Claude “hill-climbed” safety benchmarks for common misalignments like deception or sycophancy, with one constraint: it had to preserve general capabilities. We then tested its best methods on held-out benchmarks to see if they'd generalize.
Across 10 alignment failures, Claude reliably improved safety scores without degrading capabilities. Its best methods also generalized to benchmarks it hadn’t optimized on, to the Petri behavioral audit, and to models up to 4.7x larger.
Could a model one day align its stronger successors? As a first test, we had Sonnet 5 post-train an early checkpoint of Opus 4.8, a more capable model. It reached safety scores approaching those of production Opus 4.8, which went through our full alignment training.
Context
Anthropic reports that Claude could research, train and test changes that improved other models' measured safety. Its announcement describes giving the automated researcher 48 hours and one GPU for small-model experiments. The work addressed ten failure categories, with methods rejected if they harmed the capabilities being monitored.
The useful result was transfer beyond the tests used during development. According to the research article, leading methods also improved withheld evaluations and an adversarial conversational audit. This supports automating parts of safety experimentation. The authors identify a remaining measurement problem: rare failures and capabilities outside their chosen tests could escape detection.
- alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
Checking sign-in…
Loading comments…