[NEW RESEARCH] The more AI knows you, the more it copies your mistakes.
Our research team tested how personalization and memory features affect model accuracy across frontier models. What they found is worth paying attention to. Without those features, models answered correctly. Add them back in, and accuracy dropped by as much as 71%.
When a model has context on your past preferences and beliefs, it starts treating them like ground truth. It stops pushing back and starts going along with you. In finance or healthcare, where accuracy is non-negotiable, that's a serious problem. Full findings:
Switching on personalization and memory made frontier models defer to a user's stored beliefs rather than correct them, with reported accuracy losses up to 71%. Worth testing before memory goes into a high-stakes assistant.
articleMemArenaJiadong Zhang, Xiaosong Ma
articleFinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM AgentsBen Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang
articleSetoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous DataLingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou