StateBench: Testing how language models track state changes over long documents

Ai2's StateBench asks how well language models keep track of things that change over a long document, and its answer separates easy cases from hard ones. Finding the latest value of something is not where models struggle: the frontier models Ai2 tested, Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna, did well when the task was to recover a value or its most recent update. When each update depends on what came before, they broke down as inputs grew, falling to 13% accuracy or lower on the longest examples of the hardest level, according to Ai2's announcement. The same benchmark shows recurrent and hybrid architectures, which carry a running memory through the text, learning those harder levels when trained on them directly, while standard transformers never learned the hardest one.

The benchmark grades state tracking by the kind of update involved, using programming as the analogy. At the easiest level a value is set once and never changes, like a birth date. At the next, it is reassigned and only the latest setting matters, like a thermostat. The third level adds updates built on the current value, such as x = x + 5 or a running game score, so every change has to be carried forward. The hardest adds assignment by reference, such as x = y or swapping two variables, like following a ball under five cups as they are repeatedly shuffled. Ai2 builds the four versions from one sequence of assignments, keeping the number of variables and the positions of updates the same and changing only the type of assignment, so a drop in score can be traced to the type of update rather than to a longer or busier input. The levels come from Krohn–Rhodes theory, a branch of algebra that breaks state-tracking problems into basic building blocks, and the technical report by Jackson Petty, William Merrill and Ashish Sabharwal sets out the theory behind them.

The architecture question is where the work is most specific. Transformers, the design behind most current language models, have known theoretical limits on long chains of dependent updates, which has encouraged recurrent designs such as Gated DeltaNet and hybrids that mix the two. Open pretrained models, including Ai2's 7-billion-parameter Olmo 3 transformer and its Olmo Hybrid, stayed at chance on the hardest level regardless of design. When Ai2 trained small 275-million-parameter models from scratch on sequences of up to 256 assignments and tested them on up to 64,000, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, and one Gated DeltaNet variant stayed near perfect on the hardest level to about ten times its training length. Transformers degraded on longer sequences and never learned the hardest level. Ai2 reads this as a training gap for the recurrent and hybrid models and an architectural limit for transformers.

Several limits bound that reading. The direct-training results come from small models trained on the benchmark itself, which shows what each design can learn, not what a pretrained product will do. Format also mattered: some small models solved an easy problem in one presentation and sat near chance on the same problem framed as spreadsheet edits. StateBench assumes a fixed set of variables, while real conversations keep introducing new people and objects, and Ai2 lists open questions about where one Gated DeltaNet variant sits in the hierarchy. This explanation draws on the announcement and the report's abstract; the full paper's per-model scores were not read.