How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

Title:How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

Abstract:We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth $t$ grows jointly with width $n$. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large $t$. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth--width scaling regimes that govern signal propagation: (i) a \emph{subcritical regime} $t=o(\sqrt n)$, in which the infinite-width approximation remains valid; (ii) a \emph{critical regime} $t\sim c\sqrt n$, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a \emph{supercritical regime} $t\gg \sqrt n$, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results show that what constitutes "large width" depends crucially on the architecture: recurrent weight sharing makes finite-width effects visible at substantially shorter depths than in feedforward networks.

Machine Learning (cs.LG)

arXiv:2605.05113 [cs.LG]

(or arXiv:2605.05113v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.05113 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.