AgentMemBench compares five memory strategies under identical conditions across three dialogue datasets: in-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → windowing, external key-value store, graph episodic memory, compression summaries, and web-augmented memory.
External key-value retrieval dominates every quality axis (macro Recall@5 0.792) - and on truly long horizons it is the ONLY strategy that works: LoCoMo Recall@5 0.573 vs near-zero for windows, summaries, and entity graphs.
The advantage has an explicit cost: EKV consumes ~5,100 tokens of footprint vs ~300 for windowing - a direct accuracy-efficiency trade-off to budget for.
Published systems MemGPT/Letta and HippoRAG were run in the same agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition →, and all code and artifacts are released for reproduction.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Why it matters
AgentMemBench shows external key-value retrieval substantially outperforms in-context windowing, graph-based episodic memory, summarization, and web-augmented approaches on long-range recall tasks, at the cost of memory footprint — a concrete tradeoff to weigh when designing AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → memory.
Key quotes
“EKV dominates on every quality axis (macro Recall@5 0.792, MRR 0.677, F1 0.156, Faithfulness 0.354)”
“long-range recall is decisive: on LoCoMo, where the gold turn lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 <= 0.005) while EKV alone reaches 0.573”
