aiqbench analysis
This research advances the theoretical understanding of modern deep-learning optimizers like Adam and Muon by framing them as second-moment stochastic approximation methods. By deriving concrete convergence bounds, the study clarifies how estimator biases affect the neighborhood of convergence for these algorithms. This work provides essential mathematical rigor for practitioners seeking to optimize training dynamics in large-scale AI systems.
Second-Moment Stochastic Approximation Methods
arXiv:2609.36600v1 Announce Type: cross Abstract: Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.
Abstract:Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.
Optimization and Control (math.OC); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2609.36600 [math.OC]
(or arXiv:2609.36600v1 [math.OC] for this version)
https://doi.org/10.48550/arXiv.2609.36600 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
arXiv · Sep 1, 2026
Universal Dilation of Linear Itô SDEs: Quantum Trajectories and Lindblad Simulation of Second Moments
Quantum Journal · Oct 1, 2026
A Bayesian formulation of hybrid quantum-classical dynamics
arXiv · Aug 24, 2026
Quantum algorithms cut Monte Carlo sampling in simulations
Quantum Zeitgeist · Oct 1, 2026