Intel / article

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

Source: arxiv.org
Author: Shweta Mishra, Shashank Mishra
Date: Published Sep 9, 2026

Context

Finding the right memory does not ensure an will use it. MERIT tests this by giving agents tasks that require facts from earlier interactions, then checking what their tool actions actually accomplish. Its synthetic scenarios also change facts over time, making stale information a possible source of mistakes.

In one diagnostic set, the latest value appeared in retrieved memory in 55 cases, but the agent acted on it in only 30. The study therefore identifies two separate things to evaluate: whether memory supplies the right information and whether that information changes the resulting action. Retrieval accuracy alone misses the second failure.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

Key quotes

“memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00”

“embedding retrieval collapses unpredictably (0.30-0.95 across models; max seed gap 0.45), and agents act on a correctly retrieved value only 55% of the time”

“Swapping a memory's implementation moves task success by up to 60 points, and full replay is never economical”

Read the source arxiv.org