analiza aiqbench
This research provides a rigorous mathematical framework for understanding gradient descent dynamics in the Edge of Stability regime, linking persistent oscillations to improved generalization. By deriving a continuous-time mean-fluctuation model and establishing connections to Wasserstein-2 gradient flows, the study offers new theoretical tools for analyzing overparametrized networks. These insights are critical for advancing the theoretical foundations of deep learning optimization and stability analysis.
Mean--Fluctuation Dynamics at the Edge of Stability
arXiv:2606.05326v2 Announce Type: replace-cross Abstract: We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the trajectory, which has been linked to better generalization performance. We introduce the mean--fluctuation dynamics, a tractable continuous-time model coupling the window-averaged trajectory to its fluctuation covariance. Among our contributions, we rigorously derive this model from gradient descent in a sharp-valley framework, characterize its stationary states and their linear stability, and establish precise connections with other effective dynamics. Numerical experiments illustrate these predictions and their finite-time limitations. We also study our model in the overparametrized regime of wide two-layer networks at a fixed learning rate, where we rigorously derive a kinetic equation describing weights and their fluctuations as a Wasserstein-2 gradient flow, for which we prove well-posedness, a mean-field limit, and conditional convergence results.
Abstract:We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the trajectory, which has been linked to better generalization performance. We introduce the mean--fluctuation dynamics, a tractable continuous-time model coupling the window-averaged trajectory to its fluctuation covariance. Among our contributions, we rigorously derive this model from gradient descent in a sharp-valley framework, characterize its stationary states and their linear stability, and establish precise connections with other effective dynamics. Numerical experiments illustrate these predictions and their finite-time limitations. We also study our model in the overparametrized regime of wide two-layer networks at a fixed learning rate, where we rigorously derive a kinetic equation describing weights and their fluctuations as a Wasserstein-2 gradient flow, for which we prove well-posedness, a mean-field limit, and conditional convergence results.
Optimization and Control (math.OC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Mathematical Physics (math-ph); Analysis of PDEs (math.AP)
arXiv:2606.05326 [math.OC]
(or arXiv:2606.05326v2 [math.OC] for this version)
Text integral sub licența deschisă a publicației · 164 cuvinte
Subiecte conexe
Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network
arXiv · 6 iun. 2026
Designing Production-Ready Battery Energy Storage Systems for AI Factories
NVIDIA · 10 iun. 2026
Researchers Simulate Frustrated Networks Avoiding Glassy Behaviour
Quantum Zeitgeist · 10 sept. 2026
The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
arXiv · 2 sept. 2026