analiza aiqbench
This research demonstrates that restricting language models to isolated contexts prevents them from relying on spurious shortcuts, thereby improving out-of-distribution generalization and true reasoning capabilities. The study highlights a critical distinction between standard Chain-of-Thought approaches and recursive architectures, showing that simplicity bias can lead models to fail when training distributions shift. This finding offers significant theoretical insights for advancing robust AI systems beyond mere distributional accuracy.
Recursive Language Models Generalize Out of Domain
arXiv:2609.20831v1 Announce Type: new Abstract: We study when limiting what a language model can see improves learning. We compare standard CoT, the more general learner that reads the full trace, with recursive language models, which restricts itself by solving each subtask in an isolated context. In-distribution, this generality comes for free: CoT can efficiently simulate the recursive rule, so the IID generalization guarantee changes only by a constant factor, and recursion does not offer much. But out of domain, CoT can fit training by relying on context outside the current subtask, i.e. a shortcut that breaks once those tokens change; recursive context isolation rules out this failure mode. Even though CoT's class still covers the recursive rule, simplicity bias picks the shortcut over the truth. Thus, to go beyond distributional accuracy and truly reason, covering the right rule is not enough; this contrasts with classical learning theory.
Abstract:We study when limiting what a language model can see improves learning. We compare standard CoT, the more general learner that reads the full trace, with recursive language models, which restricts itself by solving each subtask in an isolated context. In-distribution, this generality comes for free: CoT can efficiently simulate the recursive rule, so the IID generalization guarantee changes only by a constant factor, and recursion does not offer much. But out of domain, CoT can fit training by relying on context outside the current subtask, i.e. a shortcut that breaks once those tokens change; recursive context isolation rules out this failure mode. Even though CoT's class still covers the recursive rule, simplicity bias picks the shortcut over the truth. Thus, to go beyond distributional accuracy and truly reason, covering the right rule is not enough; this contrasts with classical learning theory.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2609.20831 [cs.CL]
(or arXiv:2609.20831v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.20831 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 238 cuvinte
Subiecte conexe
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · 14 sept. 2026
Academia is for Ambition — Alex Zhang, MIT
Latent Space · 2 oct. 2026
[AINews] not much happened today
Latent Space · 18 sept. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026