Intel / article

AgentMemBench

Source: arxiv.org
Author: Ahmed Cherif
Date: Published Aug 4, 2026

Context

AgentMemBench found a clear trade-off between remembering older conversation details and the amount of text a memory system retained. Across 491 annotated question turns, its external key-value memory implementation led the four other tested strategies on retrieval and answer-quality measures. On the long-conversation dataset, it retrieved the relevant turn among its top five results in 57.3% of cases, while the other tested strategies achieved at most 0.5%. Its reported footprint was about 5,100 , compared with roughly 300 for recent- and web-augmented approaches. The advantage therefore came with substantially more stored text in this setup, which used Qwen2.5-7B-Instruct for generation and judging. Benchmark abstract.

Key takeaways · AI-distilled

AgentMemBench compares five memory strategies under identical conditions across three dialogue datasets: in-context windowing, external key-value store, graph episodic memory, compression summaries, and web-augmented memory.
External key-value retrieval dominates every quality axis (macro Recall@5 0.792) - and on truly long horizons it is the ONLY strategy that works: LoCoMo Recall@5 0.573 vs near-zero for windows, summaries, and entity graphs.
The advantage has an explicit cost: EKV consumes ~5,100 tokens of footprint vs ~300 for windowing - a direct accuracy-efficiency trade-off to budget for.
Published systems MemGPT/Letta and HippoRAG were run in the same , and all code and artifacts are released for reproduction.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Key quotes

“EKV dominates on every quality axis (macro Recall@5 0.792, MRR 0.677, F1 0.156, Faithfulness 0.354)”

“long-range recall is decisive: on LoCoMo, where the gold turn lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 <= 0.005) while EKV alone reaches 0.573”

Read the source arxiv.org