Tool Use Reduces Depth-Induced Collapse in OOD Reasoning

arXiv:2602.21061v2 Announce Type: replace Abstract: Many current paths to more advanced AI depend on the assumption that large language models (LLMs) can generalize learned relationships to solve complex, out-of-distribution (OOD) problems. However, this is not an easy quality to measure. For most real-world problems and benchmarks it suffices to exploit a few memorized subproblems or a small fraction of the available data to produce a correct solution. This is interpolation. Generalization requires the capacity to make use of all available data to solve problems without these shortcuts. To test this quality, we introduce a synthetic Boolean circuit reconstruction benchmark over $GF(2)$. We use an adversarial sampling oracle to block partial-information shortcuts, ensuring that each step requires the integration of all historical context with new evidence. Our evaluations reveal that standalone LLMs fall significantly short of sustained reasoning: as problem depth increases, both small and frontier models exhibit a catastrophic collapse in their ability to accurately predict the next logical step. However, we also demonstrate that tool synthesis provides a remedy to this depth-induced reasoning collapse. When allowed to generate, execute, and iteratively refine code, even small architectures can sustain accurate reasoning over long horizons and rival frontier model performance on this task. These results indicate that synthesizing tools plays a crucial role in generalization, beyond the conventional view of merely augmenting systems with specific capabilities.

Abstract:Many current paths to more advanced AI depend on the assumption that large language models (LLMs) can generalize learned relationships to solve complex, out-of-distribution (OOD) problems. However, this is not an easy quality to measure. For most real-world problems and benchmarks it suffices to exploit a few memorized subproblems or a small fraction of the available data to produce a correct solution. This is interpolation. Generalization requires the capacity to make use of all available data to solve problems without these shortcuts. To test this quality, we introduce a synthetic Boolean circuit reconstruction benchmark over $GF(2)$. We use an adversarial sampling oracle to block partial-information shortcuts, ensuring that each step requires the integration of all historical context with new evidence. Our evaluations reveal that standalone LLMs fall significantly short of sustained reasoning: as problem depth increases, both small and frontier models exhibit a catastrophic collapse in their ability to accurately predict the next logical step. However, we also demonstrate that tool synthesis provides a remedy to this depth-induced reasoning collapse. When allowed to generate, execute, and iteratively refine code, even small architectures can sustain accurate reasoning over long horizons and rival frontier model performance on this task. These results indicate that synthesizing tools plays a crucial role in generalization, beyond the conventional view of merely augmenting systems with specific capabilities.

Artificial Intelligence (cs.AI)

arXiv:2602.21061 [cs.AI]

(or arXiv:2602.21061v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2602.21061 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning

Related coverage