FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
Source
Ben Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang
Author
Ben Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang
Date
Terms in this piece · Glossary
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Anyone building an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → memory layer is likely reaching for summarization by default; this shows summarization silently drops the preference signal personalization depends on, and that plain retrieval can outperform it.
Key quotes
“Large language model (LLM) agents are increasingly used as personalized assistants in high-stakes domains such as financial advising, yet it remains unclear whether they can maintain and update an individualized user model over long horizons.”
“On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds approximately 0.47 overall accuracy or approximately 39% on multiple-choice questions.”
“Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.”
articleSetoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous DataLingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou
articleWhen Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM AgentsShweta Mishra, Shashank Mishra
articleMemArenaJiadong Zhang, Xiaosong Ma