Changyu Du, Alexander Vosseler, Filippo Mazza, Andr\'e Borrmann
Author
Changyu Du, Alexander Vosseler, Filippo Mazza, Andr\'e Borrmann
Date
Why it matters
If you are building agents that must remember a project across many sessions, this quantifies how badly off-the-shelf vector, graph and file-based memory degrade once recall has to be joined against structured domain data — best system under 33% accuracy.
It gives you a reason and a template to evaluate memory on your own domain rather than trusting generic conversational-recall scores.
A benchmark for evaluating long-term memory in LLM-based agents working on Building Information Modelling (BIM) tasks, containing 143 multi-session tasks across 19 projects and 4,016 prior sessions that require combining remembered conversational context with live IFC model queries.
Testing vector-, graph-, and file-based memory systems shows the best achieves only 32.4% answer accuracy under realistic conditions, revealing that general-purpose memory systems store project knowledge as fragmented facts rather than domain-aware representations linking conversations, project knowledge, and structured model entities.
Transcript
arXiv:2607.26072v1 Announce Type: cross Abstract: Long-term memory is becoming a core capability of LLM-based agents, but existing evaluations largely test conversational recall in open-domain or persona-grounded settings. We argue that a stronger test is whether an agent can reuse information from prior sessions while acting over a live, structured, domain-specific environment. We study this problem in Building Information Modelling (BIM), a professional engineering workflow where agents must query large IFC models while also relying on project specifications, client decisions, and engineering conventions often discussed in conversation but absent from the model. We introduce IFCMemoryBench, a benchmark for evaluating long-term memory in LLM-based BIM information retrieval. IFCMemoryBench contains 143 multi-session tasks across 19 projects and 4,016 prior sessions, derived from incomplete-information questions in IFC-Bench v2. Each task seeds missing project context across earlier conversations and later asks a probe question that can be answered only by combining remembered context with live IFC queries. Our evaluation framework decomposes memory performance into ingestion, retrieval, and utilization, and measures both answer quality and memory quality with expert-validated LLM judges. We evaluate representative vector-, graph-, and file-based memory systems. The strongest system achieves only 32.4% answer accuracy under a deployment-realistic ingestion scope, and remains below 60% under oracle-filtered ingestion or a stronger probe agent. Analysis shows that current general-purpose memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts. These results reveal a domain-transfer gap in agent memory and suggest that reliable professional agents require domain-aware memory representations linking conversations, project knowledge, and structured model entities.
