analiza aiqbench
This research establishes that stochastic gradient descent with a constant stepsize exhibits fundamentally different long-term behavior at flat minima compared to strongly convex landscapes, converging to non-Gaussian stationary distributions. The study provides rigorous bounds on convergence rates dependent on the local flatness exponent and characterizes the scaling limits of iterates under Markovian noise. These theoretical insights are critical for understanding optimization dynamics in deep learning models where loss surfaces typically feature wide, flat basins rather than sharp quadratic minima.
Scaling Limits of Constant-Stepsize SGD at Flat Minima
arXiv:2607.16384v1 Announce Type: new Abstract: For stochastic gradient descent (SGD) with a constant stepsize $\alpha$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrt{\alpha}$ scaling and a Gaussian limit as $\alpha\downarrow 0$. We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize $\alpha$, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an $\alpha$-dependent metric. When the minimizer $x_\star$ has local flatness exponent $m\ge2$, meaning that $\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d$ as $x\to x_\star$, we obtain a contraction bound with factor $1-c\alpha^{m-1}$, where $c>0$ is a constant. This recovers the factor $1-c\alpha$ in the quadratic case $m=2$. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale $\alpha^{1/m}$ and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation $$ dY_t=-h_0(Y_t)\,dt+\Sigma^{1/2}\,dB_t , $$ where $h_0$ is the limiting drift at the minimizer and $\Sigma$ denotes the asymptotic covariance. This recovers the Gaussian limit when $m=2$ and gives generally non-Gaussian stationary limits in the flat case $m>2$. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.
Abstract:For stochastic gradient descent (SGD) with a constant stepsize $\alpha$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrt{\alpha}$ scaling and a Gaussian limit as $\alpha\downarrow 0$. We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize $\alpha$, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an $\alpha$-dependent metric. When the minimizer $x_\star$ has local flatness exponent $m\ge2$, meaning that $\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d$ as $x\to x_\star$, we obtain a contraction bound with factor $1-c\alpha^{m-1}$, where $c>0$ is a constant. This recovers the factor $1-c\alpha$ in the quadratic case $m=2$. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale $\alpha^{1/m}$ and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation $$ dY_t=-h_0(Y_t)\,dt+\Sigma^{1/2}\,dB_t , $$ where $h_0$ is the limiting drift at the minimizer and $\Sigma$ denotes the asymptotic covariance. This recovers the Gaussian limit when $m=2$ and gives generally non-Gaussian stationary limits in the flat case $m>2$. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.
Machine Learning (cs.LG); Optimization and Control (math.OC); Probability (math.PR); Machine Learning (stat.ML)
arXiv:2607.16384 [cs.LG]
(or arXiv:2607.16384v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.16384 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 337 cuvinte
Subiecte conexe
SPIR-V on ROCm: A Portable IR for AMD GPUs
AMD · 20 iul. 2026
Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds
arXiv · 9 sept. 2026
Steady-State Convergence of Stochastic Approximation
arXiv · 15 sept. 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · 21 iul. 2026