When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents
Source
arxiv.org
Author
Wenhui Chu (University at Albany, State University of New York)
Date
Why it matters
Before building a memory layer for an agent, compare it with simply re-reading current sources. In this study, source-filtered re-reading was the cheapest option even at about 10,000 tokens.
Key takeaways · AI-distilled
The study asks whether an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → should repair memory or re-read sources when documents behind its derived facts are revoked or replaced, using medication- and problem-list tasks from public ICU records and two 7B models.
Costs were counted over the whole pipeline (ingest, revision and every use), with memory supplied in full rather than retrieved. On that basis every memory pipeline cost at least twice full re-reading on short records in held-out conditions.
In a small development sweep that padded records to about 10,000 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →, memory became cheaper than full re-reading after 2-14 uses, partly because extraction truncated, yet source-filtered re-reading stayed cheapest.
None of the four primary confirmatory tests in the replacement study reached statistical significance, so the author frames the result as a call to benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → any memory layer against source-filtered re-reading.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleWhen Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM AgentsShweta Mishra, Shashank Mishra
articleRENDER: Controlling Reader-Facing Evidence in LLM Memory EvaluationYuan Si, Simeng Han, Daming Li, Jialu Zhang
articleAssurance Envelopes for Autonomous Coding Agents: Minimum-Cost Evidence for Software ChangeAnjan Goswami