MemArena is a single-world memory benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →: 50 simulated agents interacting for 15 days (10.3M dialog tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →) with co-generated ground truth across recall, reasoning and trustworthiness dimensions.
Memory-backend choice beats reader scaling for content accuracy: at Qwen3-0.6B, swapping Memobase for MemSearch gains +32.5/+19.2pp - more than scaling the reader model itself (+10.6/+6.8pp).
Permission-aware access fails universally: oracle retrieval leaks private information heavily while every other backend is too timid to disclose even permitted facts.
Memory-search latency is a fixed, moderate 7-87ms on an edge node - a small share of time-to-first-token for most reader-backend combinations, so latency is not the blocker.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
If you're building on-device personal memory assistants, backend design choices matter more than swapping in a bigger model, and current permission-aware privacy access mechanisms are broken across the board—both are load-bearing facts for architecture decisions.
Key quotes
“Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose.”
“Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase-to-MemSearch gains +32.5/+19.2 pp, exceeding MemSearch reader scaling (+10.6/+6.8 pp).”
articleAgentMemBenchAhmed Cherif
articleSetoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous DataLingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou
articleFinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM AgentsBen Wang, Kang Zhou, Lifan Guo, Feng Chen, Chi Zhang