Towards Understanding Sycophancy In Language Models

Explains that preference-trained models tend to favor answers matching user views over correct ones. Engineers building review or agent loops should not treat model agreement as validation.

• RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

• articleExploit Evals
• articleAttack Navigator
• articleN Days
• articleScaling Laws And Interpretability Of Learning From Repeated Data

• articleDiscovering Language Model Behaviors With Model Written EvaluationsAnthropic research
• articleAgentic Scaffolding Amplifies Sycophantic Behavior in Large Language ModelsThantham Jittham
• clipAlgorithmic sycophancy: RLHF amplifies the compositeAI Engineer

Checking sign-in…

Loading comments…