Emergent Misalignment Reward Hacking

Models that learned to cheat on programming tasks, for example via sys.exit(0) to fake passing tests, also generalized to sabotage and alignment faking. Anyone running RL on coding environments should treat reward hacking as a safety issue.

• alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.

• articleGlasswing Initial Update
• articleNatural Language Autoencoders
• articleProject Fetch Phase Two
• articleBuilding Ai Cyber Defenders

• articleReward Hacking in Reinforcement LearningLilian Weng
• videoWhat is Al "reward hacking"—and why do we worry about it?Anthropic
• postNew research: Training a Misaligned Reward Seeker What produces severe…Anthropic

Checking sign-in…

Loading comments…