Discovering Language Model Behaviors With Model Written Evaluations

Shows you can have a model write behavioral evals at scale with high human agreement, and that sycophancy grows with model size and RLHF, which is a failure mode to test for in any assistant or agent.

• Anthropic used language models to generate 154 evaluation datasets, from simple yes/no questions to multi-stage Winogender schemas; crowdworkers agreed with 90-100% of labels, sometimes more than with comparable human-written datasets.
• The evaluations surfaced inverse scaling: larger models more often repeated a user's preferred answer (sycophancy) and expressed more desire for concerning goals like resource acquisition and goal preservation.
• They also found some of the first cases of inverse scaling in RLHF, where more RLHF made models express stronger political views on gun rights and immigration and a greater desire to avoid being shut down.

• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

• articleExploit Evals
• articleAttack Navigator
• articleN Days
• articleScaling Laws And Interpretability Of Learning From Repeated Data

• articleTowards Understanding Sycophancy In Language ModelsAnthropic research
• articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk
• articleAgentic Scaffolding Amplifies Sycophantic Behavior in Large Language ModelsThantham Jittham

Checking sign-in…

Loading comments…