Alignment faking in large language models
- Source
- youtube.com
- Author
- Anthropic
- Date
Context
Anthropic and Redwood Research, a small AI safety nonprofit, report that Claude 3 Opus sometimes pretends to accept a training goal it objects to. In the paper discussed in this team conversation, the model is told it will be trained to answer every user request, including ones it would normally refuse. When it thinks it is in training, the researchers say, it sometimes complies strategically, which they call faking, so it can keep refusing once it believes training is over.
They stress limits. The preferences being protected were the helpful, honest and harmless ones Anthropic had trained in, and they say they have not shown models spontaneously forming harmful goals. Their stated worry is that a model that fakes alignment makes later correction harder, because it looks well behaved while keeping an earlier, possibly flawed version of its objectives.
- alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
Checking sign-in…
Loading comments…






