Memory and personalization made frontier models agree with your errors
- Source
- x.com
- Date
[NEW RESEARCH] The more AI knows you, the more it copies your mistakes. Our research team tested how personalization and memory features affect model accuracy across frontier models. What they found is worth paying attention to. Without those features, models answered correctly. Add them back in, and accuracy dropped by as much as 71%. When a model has context on your past preferences and beliefs, it starts treating them like ground truth. It stops pushing back and starts going along with you. In finance or healthcare, where accuracy is non-negotiable, that's a serious problem. Full findings:




Context
A memory system can preserve a user's mistaken belief while losing the correction that followed it. WRITER's MIST study tested synthetic conversations containing plausible misconceptions, then compared answers using extracted memories with answers using the full conversation. Across the tested model families, memory increased agreement with the misleading . Follow-up experiments implicated the extracted content: retaining assistant responses or using conversation summaries reduced this tendency. The finding concerns how conversation history is represented, with corrective context providing a useful counterweight to remembered user claims.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleMemArenaJiadong Zhang, Xiaosong Ma
articleFinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM AgentsBen Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang
articleThe Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory AgentsJundong Hu, Shekar Ramachandran
Checking sign-in…
Loading comments…



