Intel / article

MemArena

Source: arxiv.org
Author: Jiadong Zhang, Xiaosong Ma
Date: Published Aug 5, 2026

Context

Remembering a conversation and deciding whether to reveal its contents are separate challenges for personal assistants. MemArena tests both using a simulated world with fifty agents interacting over fifteen days. It generates conversation histories and corresponding answers, allowing researchers to test memory systems against connected events rather than isolated questions. Benchmark description.

The revised study reports two failures in permission-aware access: its oracle retrieval condition leaked protected information, while other memory systems failed to surface the protected fact. These outcomes show why retrieving relevant information is insufficient to establish appropriate disclosure. The evidence comes from the tested systems in a simulated conversational world. Revised findings.

Key takeaways · AI-distilled

MemArena is a single-world memory : 50 simulated agents interacting for 15 days (10.3M dialog ) with co-generated ground truth across recall, reasoning and trustworthiness dimensions.
Memory-backend choice beats reader scaling for content accuracy: at Qwen3-0.6B, swapping Memobase for MemSearch gains +32.5/+19.2pp - more than scaling the reader model itself (+10.6/+6.8pp).
Permission-aware access fails universally: oracle retrieval leaks private information heavily while every other backend is too timid to disclose even permitted facts.
Memory-search latency is a fixed, moderate 7-87ms on an edge node - a small share of time-to-first-token for most reader-backend combinations, so latency is not the blocker.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Key quotes

“Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose.”

“Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase-to-MemSearch gains +32.5/+19.2 pp, exceeding MemSearch reader scaling (+10.6/+6.8 pp).”

Read the source arxiv.org