Intel / article

IFCMemoryBench

Source: arxiv.org
Author: Changyu Du, Alexander Vosseler, Filippo Mazza, Andr\'e Borrmann
Date: Published Jul 30, 2026

Context

An engineering assistant may need both a building's digital model and decisions recorded in earlier conversations to answer a project question. IFCMemoryBench tests that combination across 143 tasks from 19 projects, placing missing information in earlier sessions and requiring later answers to combine it with queries of structured building data.

The authors report that the strongest tested memory system achieved 32.4% answer accuracy when processing prior conversations under their deployment-realistic setup. Systems often found relevant discussion but retained incomplete project facts. That points to a specific weakness: finding a related conversation does not necessarily recover enough precise information to answer an engineering question.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“The strongest system achieves only 32.4% answer accuracy under a deployment-realistic ingestion scope, and remains below 60% under oracle-filtered ingestion or a stronger probe agent.”

“Analysis shows that current general-purpose memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts.”

Read the source arxiv.org