StateBench: Testing how language models track state changes over long documents
Source
allenai.org
Date
Why it matters
Long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → state tracking underlies code reading and AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → memory. StateBench separates easy latest-update lookups from hard chained updates, so you can see which architectures, including hybrids, actually hold up.
Key takeaways · AI-distilled
Ai2's StateBench grades state tracking in four levels that map to variable assignment: values that never change, reassignment where only the latest value counts, self-referential updates like x = x + 5, and assignment by reference such as x = y or swaps.
To compare levels fairly, Ai2 derives the three easier versions from one general finite sequence, keeping the same number of variables and distinct values and the same assignment positions, so only the type of assignment changes.
Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna did well on the two easier levels, but on the longest general finite problems their accuracy fell to 13% or lower.
Problem format changed results: 275-million-parameter hybrid and Gated DeltaNet models scored near ceiling on RULER-style r-trivial problems yet near chance when the same problems were framed as spreadsheet edits.
Trained from scratch on up to 256 assignments, Gated DeltaNet and hybrid models generalized almost perfectly on the three easier levels out to tens of thousands of assignments, while transformers never learned the hardest level.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleLong-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent ProgressZongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
articleAutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model ResearchMarjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva, Harold Chaput, Mohammad Reza Taesiri