Recursive Language Models Generalize Out of Domain
Source
Chenxiao Yang, Zhiyuan Li, David McAllester, Nathan Srebro
Author
Chenxiao Yang, Zhiyuan Li, David McAllester, Nathan Srebro
Date
Key takeaways · AI-distilled
In-distribution, isolating context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → buys almost nothing: chain-of-thoughtHaving a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.Full definition → can efficiently simulate the recursive rule, so its IID generalization guarantee only changes by a constant factor.
The specific failure mode: CoT can exploit context sitting outside the current subtask as a shortcut, and that shortcut breaks the moment those out-of-subtask tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → change under distribution shift.
The gap isn't about capacity - CoT's function class still covers the correct recursive rule - the authors attribute it to simplicity bias picking the shortcut over the truth.
The authors frame this as a departure from classical learning theory: merely having a model class that covers the right rule isn't enough for genuine out-of-domain reasoning.
Terms in this piece · Glossary
chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
subagent — A separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Argues that isolating subtasks into separate contexts, rather than one long chain-of-thought, prevents models from exploiting spurious in-context shortcuts, supporting subagentA separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.Full definition →/recursive decomposition patterns for out-of-domain robustness.
articleModular Cognitive Architecture Emerges in Large Language ModelsPengrui Han, Jacob Andreas, Evelina Fedorenko, Andrea Gregor de Varda
articleChildren, but not language models, show accelerating returns in word learningMichael C. Frank
articleGRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual SettingsKonstantin Dobler, Federico Scozzafava, Jonathan Janke, Mohamed Ali, Simon Lehnerer