analiza aiqbench
This research introduces Autoregressive Thought Flow (ATF), a novel framework that models reasoning steps as multimodal distributions within a continuous latent space rather than discrete tokens. By combining a causal autoregressive backbone with a lightweight diffusion head, the method preserves pretrained capabilities while enabling variable-length reasoning paths that capture diverse solution strategies. The approach demonstrates improved accuracy on mathematical tasks and broader coverage of solutions compared to single-prediction models.
Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space
arXiv:2609.33271v1 Announce Type: new Abstract: Reasoning problems often admit multiple valid ways to proceed. Continuous reasoning promises to move computation beyond language tokens into a more compact latent space, but representing several plausible ways to think next remains difficult. We introduce Autoregressive Thought Flow (ATF), which models the next continuous thought as a multimodal distribution. A causal autoregressive model performs the reasoning computation, while a lightweight diffusion head generates a plausible next thought from the resulting condition. The sampled thought is fed back into the model, allowing continuous reasoning to unfold for a variable number of steps while preserving the pretrained backbone. Across mathematical reasoning tasks, ATF improves accuracy with compact latent traces and benefits from reinforcement learning and additional test-time thinking. Multi-sample evaluation shows broader solution coverage, indicating that its multimodal predictions capture useful diversity among reasoning paths. Our results suggest that continuous reasoning is more effective when multiple possible next thoughts remain available rather than being collapsed into a single prediction.
Abstract:Reasoning problems often admit multiple valid ways to proceed. Continuous reasoning promises to move computation beyond language tokens into a more compact latent space, but representing several plausible ways to think next remains difficult. We introduce Autoregressive Thought Flow (ATF), which models the next continuous thought as a multimodal distribution. A causal autoregressive model performs the reasoning computation, while a lightweight diffusion head generates a plausible next thought from the resulting condition. The sampled thought is fed back into the model, allowing continuous reasoning to unfold for a variable number of steps while preserving the pretrained backbone. Across mathematical reasoning tasks, ATF improves accuracy with compact latent traces and benefits from reinforcement learning and additional test-time thinking. Multi-sample evaluation shows broader solution coverage, indicating that its multimodal predictions capture useful diversity among reasoning paths. Our results suggest that continuous reasoning is more effective when multiple possible next thoughts remain available rather than being collapsed into a single prediction.
Artificial Intelligence (cs.AI)
arXiv:2609.33271 [cs.AI]
(or arXiv:2609.33271v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.33271 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 252 cuvinte
Subiecte conexe
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Does DiffusionGemma do latent reasoning?
AI Alignment Forum · 16 aug. 2026
Academia is for Ambition — Alex Zhang, MIT
Latent Space · 2 oct. 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · 14 sept. 2026