FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents

FinPerMA tests whether an AI assistant updates its understanding of a person after events change the preferences it should remember. The research abstract describes controlled investor histories and a checkpoint after a significant event, allowing researchers to distinguish remembering earlier facts from incorporating a change.

Across 2,994 questions about 276 personas, the authors report that no configuration given the full history exceeded approximately 47% overall accuracy. Their analysis found that summaries could preserve factual details while losing preference cues, allowing simple retrieval to outperform some specialized memory systems. The finding identifies a weakness in personalization on this constructed benchmark, rather than measuring the quality of advice delivered to real investors.

“Large language model (LLM) agents are increasingly used as personalized assistants in high-stakes domains such as financial advising, yet it remains unclear whether they can maintain and update an individualized user model over long horizons.”

“On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds approximately 0.47 overall accuracy or approximately 39% on multiple-choice questions.”

“Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.”