aiqbench analysis
This paper gives a rigorous probabilistic treatment of SGD with momentum, the optimizer behind much of modern deep learning, by modeling the parameter and momentum updates as a nonlinear time series. It proves that the process forgets its starting point at a geometric rate and reaches a unique stationary regime, then derives Gaussian limits for averaged and last-iterate behavior. For ML practitioners, the main value is a theoretical basis for how learning rate and momentum choices set the variance and fluctuations of training, though the results are established in a quadratic model and under specific moment assumptions.
A Nonlinear Time-Series Theory for Constant-step SGD with Momentum
arXiv:2610.16034v1 Announce Type: cross Abstract: Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite $p$th moment for $p > 1$, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.
Abstract:Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite $p$th moment for $p > 1$, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2610.16034 [stat.ML]
(or arXiv:2610.16034v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2610.16034 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 216 words
