When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents
- Source
- arxiv.org
- Author
- Wenhui Chu (University at Albany, State University of New York)
- Date

Context
An that derives facts from documents faces a choice when a source is revoked or replaced: repair its stored memory, or re-read the current documents. A paper on arXiv, posted October 2, 2026, measures the total cost of each option, counting setup, revisions and every later use, using medication and problem lists built from public ICU records and two 7-billion-parameter models.
On short records, repairing only the affected part of the memory used 5 to 10 times fewer than rebuilding it, yet in held-out conditions every memory approach cost at least twice as much as re-reading everything. When irrelevant documents stretched records to about 10,000 tokens, memory became cheaper than full re-reading after 2 to 14 uses, partly because extraction truncated the text, and re-reading only the relevant sources stayed cheapest.
The padding test was a small pre-specified development sweep, and none of the four primary confirmatory tests in the replacement study reached statistical significance. So in this setup repair beat only rebuilding, not source-filtered re-reading.
- The study asks whether an agent should repair memory or re-read sources when documents behind its derived facts are revoked or replaced, using medication- and problem-list tasks from public ICU records and two 7B models.
- Costs were counted over the whole pipeline (ingest, revision and every use), with memory supplied in full rather than retrieved. On that basis every memory pipeline cost at least twice full re-reading on short records in held-out conditions.
- In a small development sweep that padded records to about 10,000 tokens, memory became cheaper than full re-reading after 2-14 uses, partly because extraction truncated, yet source-filtered re-reading stayed cheapest.
- None of the four primary confirmatory tests in the replacement study reached statistical significance, so the author frames the result as a call to any memory layer against source-filtered re-reading.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleWhen Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM AgentsShweta Mishra, Shashank Mishra
articleRENDER: Controlling Reader-Facing Evidence in LLM Memory EvaluationYuan Si, Simeng Han, Daming Li, Jialu Zhang
articleWhen Old Facts Return: Re-Reads, Reverts, and the Limits of Temporal MemoryNeeraj Yadav (Called It Inc.)
Checking sign-in…
Loading comments…