analiza aiqbench
This theoretical work establishes exact identities for weight norm imbalances in deep nonlinear networks, providing a rigorous framework to predict escape times from training plateaus during feature acquisition. By deriving scaling laws that depend on the number of bottleneck-initialized layers, the study bridges the gap between shallow linear analyses and complex nonlinear dynamics observed in practical training regimes. The findings offer valuable insights into the mathematical mechanisms governing sharp transitions in deep learning optimization without relying on empirical heuristics alone.
A Theory of Saddle Escape in Deep Nonlinear Networks
arXiv:2605.01288v4 Announce Type: replace Abstract: In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of $r$ layers initialized at the bottleneck scale $\varepsilon$, the escape time is $\Theta(\Gamma^{-1})$ for $r=1$, $\Theta(\Gamma^{-1}\log(1/\varepsilon))$ for $r=2$, and $\Theta(\Gamma^{-1}\varepsilon^{-(r-2)})$ for $r\geq3$, where $\Gamma=\|\mathbb{E}[yx]\|$. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.
Abstract:In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of $r$ layers initialized at the bottleneck scale $\varepsilon$, the escape time is $\Theta(\Gamma^{-1})$ for $r=1$, $\Theta(\Gamma^{-1}\log(1/\varepsilon))$ for $r=2$, and $\Theta(\Gamma^{-1}\varepsilon^{-(r-2)})$ for $r\geq3$, where $\Gamma=\|\mathbb{E}[yx]\|$. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.
Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (stat.ML)
arXiv:2605.01288 [cs.LG]
(or arXiv:2605.01288v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.01288 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 256 cuvinte
