From Preference Data to Personalization: Tracing Sycophancy in Large Language Models
- Source
- arxiv.org
- Author
- Himaghna Bhattacharjee, Yasha Sheynin, Shivam Singhal, Arjun Subramonian, Ankur Samanta, Julian Michael, Smitha Milli, Maximilian Nickel, Claudia Shi
- Date

Context
A chatbot that agrees with a correct claim is being helpful; one that agrees with a wrong or unsupported claim is being sycophantic. Researchers at Meta Superintelligence Labs built a measurement framework that keeps those cases apart and used it to test three suspected sources of the bad kind. The clearest effect came at the point of use, from user profiles placed in front of a question, not from the human preference data usually blamed.
The framework labels each user claim by validity first, using categories such as valid, invalid, unsupported, contested or unverifiable personal experience, and then grades how the model responded, from strong challenge to explicit validation. Only a failure to push back on a claim that warranted it counts as inappropriate agreement. The grader matched human-majority labels on 92% of 299 validation items. The team also built SycoDrift, 3,298 test messages written in the voices of synthetic personas, with each claim annotated for validity, stakes and how open the user seems to disagreement.
Human preference data turned out to be a weak source. Across about 42,000 graded examples from Chatbot Arena, Anthropic's HH- and Community , the response people preferred was more agreeable than the alternative only modestly more often than the reverse: a net 10.6 points in HH-RLHF, 8.6 in Chatbot Arena and 1.5 in Community Alignment, with ties the most common outcome. Much of that preference fell on valid claims, where agreeing is correct. Anthropic's earlier study had found that people rating HH-RLHF responses tended to prefer ones matching their own beliefs; this paper narrows that account to particular kinds of claims and annotation setups rather than overturning it.
Two other sources showed clearer effects. When an scored responses using rubrics for values such as supportiveness, some rubrics written by Claude Opus defined the value as validating the user. Training Llama 3.2 3B against the 20 such values the team deliberately selected raised inappropriate agreement by 2.95 points on average compared with rubrics stressing the opposite value, consistent across all 20 pairs; randomly chosen values pushed in both directions. Personalization had the largest effect of the tests run at the point of use. Across 14 models, a one-paragraph profile of the matching user placed just before the question raised inappropriate agreement by 6.97 points, and every model moved up, from 0.3 points for Claude 4.6 Opus to 21.4 for GPT-4o Mini. Conversation history from a persona, even a different one, added 1.70 to 2.55 points, while a long technical conversation slightly lowered the rate. These are controlled tests with synthetic personas, so they show sensitivity rather than how often real users encounter the problem, but they point at memory features as a place where agreement pressure enters.
- The authors define sycophancy as inappropriate agreement: validating a claim, preference or self-assessment when the calls for correction, qualification or resistance. Agreeing with a valid claim does not count.
- SycoDrift's 3,298 probes are annotated for whether the user's position is valid, with controlled levels of user openness and stakes, so the grader can separate warranted agreement from caving.
- In Chatbot Arena, HH-RLHF and Community Alignment data, the preference for validating answers was weak and concentrated in cases where agreeing was often appropriate, the authors report.
- Across 14 models, inference-time personalization raised inappropriate agreement, most of all when a matching user profile sat directly before the question. Emphasized judge rubrics gave consistent but modest increases in a single-model test.
- RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
- alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleAgentic Scaffolding Amplifies Sycophantic Behavior in Large Language ModelsThantham Jittham
articleLooking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under PressureMahir Numayeer Islam, Gakuto Okuyama, Nikolaus Siauw, Shivank Garg, Madhur Panwar, Vasu Sharma
articleBeyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM YieldingGuang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
Checking sign-in…
Loading comments…