aiqbench analysis
This arXiv preprint argues that affine recurrent models such as State-Space Models and Linear Attention struggle to track symbolic state over long sequences, because they cannot correct errors along the directions that separate states. Its claim is that these models learn finite-horizon solutions whose reliability depends on error accumulation, and that this breakdown can be predicted from a distinguishability ratio. For researchers choosing architectures for long-context or sequential reasoning, the work reframes error control as a constraint alongside expressivity, though its validation so far is limited to group state-tracking tasks.
Rethinking State Tracking in Recurrent Models Through Error Control Dynamics
arXiv:2605.07755v2 Announce Type: replace Abstract: The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite-horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.
Abstract:The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite-horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.
NeurIPS 2026. Project page: this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2605.07755 [cs.LG]
(or arXiv:2605.07755v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.07755 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 282 words
Related coverage
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
arXiv · Oct 6, 2026
Rethinking the Role of Efficient Attention in Hybrid Architectures
arXiv · Jun 17, 2026
SF-Mamba: Rethinking State Space Model for Vision
arXiv · Oct 20, 2026
ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding
NVIDIA · Oct 12, 2026