Parametric Multimodal User Memory: Storing What Captions Cannot Carry
Source
Bojie Li, Noah Shi
Author
Bojie Li, Noah Shi
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → memory built from transcripts and captions loses perceptual identity: caption retrieval recovers as little as 0.11 of an encoder's recall on voices and faces. groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → with a VLM and storing an encoder identity key as an inline token closes most of the gap.
articleVoice Memory for Agentic Speech RecognitionChao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg
articleThe Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual ReasoningGarima Arya Yadav, Nilay Yilmaz, Yezhou Yang
articleRENDER: Controlling Reader-Facing Evidence in LLM Memory EvaluationYuan Si, Simeng Han, Daming Li, Jialu Zhang