aiqbench analysis
This paper advances the theoretical understanding of STORM, a stochastic recursive momentum algorithm, by proving its convergence rates under weaker smoothness assumptions than previously required. By removing the strong average smoothness constraint, the authors demonstrate that STORM maintains optimal performance for nonconvex, convex, and strongly convex objectives using only standard hyperparameter adjustments. These findings are significant for researchers developing robust optimization methods for large-scale machine learning models where strict smoothness conditions may not hold.
Convergence Analysis of STORM Under Different Geometries
arXiv:2610.01599v1 Announce Type: cross Abstract: Stochastic recursive momentum (STORM) achieves fast convergence for nonconvex optimization via the variance reduction effect, but existing analyses rely on the strong average smoothness assumption. In this paper, we study the convergence of STORM for different objectives without average smoothness. We first revisit the results under average smoothness, obtaining the $O(T^{-1/3})$ bound for nonconvex objectives and the $O(\sigma^2/(\mu T))$ bound for last-iterate output under the $\mu$-Polyak--\L{}ojasiewicz~(PL) condition. Without average smoothness, we design an auxiliary sequence and compare the STORM update with it in the analysis. With the help of this sequence, we prove that STORM still attains an $O(T^{-1/4})$ rate for nonconvex objectives, which is optimal under standard smoothness. For convex and $\lambda$-strongly convex objectives, we further prove averaged and last-iterate bounds with optimal rates of $O(\sigma R/\sqrt T)$ and $O(\sigma^2/(\lambda T))$, respectively. All the obtained results use the same STORM recursion with different hyperparameter choices.
Abstract:Stochastic recursive momentum (STORM) achieves fast convergence for nonconvex optimization via the variance reduction effect, but existing analyses rely on the strong average smoothness assumption. In this paper, we study the convergence of STORM for different objectives without average smoothness. We first revisit the results under average smoothness, obtaining the $O(T^{-1/3})$ bound for nonconvex objectives and the $O(\sigma^2/(\mu T))$ bound for last-iterate output under the $\mu$-Polyak--Łojasiewicz~(PL) condition. Without average smoothness, we design an auxiliary sequence and compare the STORM update with it in the analysis. With the help of this sequence, we prove that STORM still attains an $O(T^{-1/4})$ rate for nonconvex objectives, which is optimal under standard smoothness. For convex and $\lambda$-strongly convex objectives, we further prove averaged and last-iterate bounds with optimal rates of $O(\sigma R/\sqrt T)$ and $O(\sigma^2/(\lambda T))$, respectively. All the obtained results use the same STORM recursion with different hyperparameter choices.
Optimization and Control (math.OC); Machine Learning (cs.LG)
arXiv:2610.01599 [math.OC]
(or arXiv:2610.01599v1 [math.OC] for this version)
https://doi.org/10.48550/arXiv.2610.01599 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 244 words
Related coverage
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
arXiv · Sep 9, 2026
The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient
arXiv · Jun 17, 2026
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muon
arXiv · Aug 28, 2026
Scale, Drivers, and Limits of Representational Alignment Across Large Language Models
arXiv · Oct 15, 2026