Parametric Multimodal User Memory: Storing What Captions Cannot Carry
Source
arxiv.org
Author
Bojie Li, Noah Shi
Date
Context
The proposed perceptual-memory system separates locating a person or sound from recognizing it. A vision-LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → first identifies the relevant image region or audio segment; a specialized encoder then turns that region into an identity representation. That identity key is stored as a single inline entry the language model reads through attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → while generating, with no external lookup. This preserves perceptual information that a written caption may discard. The authors keep exact verbal facts in a separate text store, combining recognition with remembered statements rather than requiring one representation to serve both purposes. Memory design.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleWhen Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM AgentsShweta Mishra, Shashank Mishra
articleThe Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory AgentsJundong Hu, Shekar Ramachandran
articleVoice Memory for Agentic Speech RecognitionChao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg