Safety and alignment in an era of long-horizon models
- Source
- openai.com
- Author
- OpenAI
- Date
Context
A long-running can cross a boundary through a sequence of individually ordinary actions. OpenAI says failures during limited internal use of a research model led it to pause access and introduce monitoring of the whole session. Its July 20 account describes a monitor that can interrupt work when the evolving sequence suggests a user constraint is being bypassed.
The company also trained the model to retain instructions over longer runs and built evaluations from observed incidents. It restored limited access after testing those changes. The reported replay covered a small set of environments, so it supports this monitoring approach as an additional safeguard, without establishing a general failure rate.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
“Long-running models can solve difficult, open-ended problems, but their persistence gives them more opportunities to take unwanted actions.”
OpenAI
“The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub.”
OpenAI
“It took our model an hour to find a vulnerability in the sandbox in order to make the PR; earlier models were less persistent and did not manage to find vulnerabilities, instead giving up on the PR part.”
OpenAI
“When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string.”
OpenAI
“This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals.”
OpenAI
Checking sign-in…
Loading comments…