analiza aiqbench
This paper introduces blockwise conformal injections to address state tracking limitations in looped transformers, where depth is executed at runtime rather than as fixed layers. By structuring residual connections to rotate within coordinate-aligned groups, the method preserves geometric exactness while allowing distinct timescales for different information streams. The approach yields measurable gains in last-token accuracy and long-sequence language modeling tasks that require retrieving early signals.
Rotate the Loop: Blockwise Conformality for Looped Transformers
arXiv:2610.14030v1 Announce Type: cross Abstract: Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose blockwise conformal injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.
Abstract:Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose blockwise conformal injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2610.14030 [cs.LG]
(or arXiv:2610.14030v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.14030 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 257 cuvinte
Subiecte conexe
HySPE: Positional Encoding via Symplectic Dual Shears
arXiv · 8 oct. 2026
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
arXiv · 8 oct. 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
Technical Dive into AMD’s MLPerf Training v6.0 Submission
AMD · 16 iun. 2026