aiqbench analysis
This research proposes a novel recurrent architecture that combines logarithmic depth with linear runtime, offering a viable alternative to standard Transformers for long-sequence language modeling. By extending balanced-tree recursive operators to autoregressive prediction, the model achieves robust length extrapolation while maintaining performance comparable to ALiBi-based approaches. This work addresses critical scalability limitations in current deep learning models without sacrificing parallelization efficiency.
Log-Depth Recurrent Language Modeling
arXiv:2609.28212v1 Announce Type: cross Abstract: Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.
Abstract:Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.28212 [cs.LG]
(or arXiv:2609.28212v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.28212 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 189 words
Related coverage
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026
CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models
arXiv · Sep 3, 2026
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
arXiv · Oct 12, 2026
Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
arXiv · Oct 9, 2026