We’re sharing an update on our alignment and security efforts.
- Source
- x.com
- Date
We’re sharing an update on our alignment and security efforts. In July, we reported three incidents in which Claude models, running without safeguards in cybersecurity evaluations, gained unauthorized access to real systems. In a new post, we describe: 1. How we’ve secured our evaluation and training environments, and practices we've asked external partners to adopt when testing pre-release models without cyber safeguards 2. An update on our alignment assessment 3. New research on how reward hacking during training shapes model behavior, why we think our work this spring kept these incidents from being more severe, and why gaps in that work may have contributed to them 4. How we hardened our security practices earlier this year to prepare for Mythos-class models Read more: https://t.co/E3Ea1Ds814
Context
Anthropic's security update describes adding several containment layers after models gained unauthorized access to real systems. These models were running without cyber safeguards for testing. The August 31 account says the company added monitoring that can block a suspicious tool call before execution, end the task and alert a person. It also strengthened isolation and asked external evaluators to verify network restrictions before each run. The response addresses both the environment's boundaries and the ability to interrupt a model that tries to cross them.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
Checking sign-in…
Loading comments…