aiqbench analysis
This research reveals that the second-moment memory inherent in Adam optimizers can actively suppress convergence rates, even when gradient noise exhibits finite variance. By establishing a theoretical lower bound on optimization progress relative to this memory length, the study provides critical insights for tuning hyperparameters like $\beta_2$ in large-scale training scenarios.
Second-Moment Memory in Coordinatewise Adam
arXiv:2608.15824v1 Announce Type: new Abstract: Adam retains a moving average of past squared gradients in its denominator, but the optimization cost of this memory is not well understood. We show that second-moment memory can itself suppress progress toward the optimum even under finite-variance stochastic gradients. For a simple two-point oracle, the expected positive normalized update is $O(M_2^{-1/2})$ after an initialization transient, where $M_2=(1-\beta_2)^{-1}$ is the second-moment memory length. We convert this directional bound, under the stated memory and stepsize scaling, into an average-stationarity lower bound of the same order on a smooth convex problem with normalized gap, smoothness, and variance. Long second-moment memory can slow optimization even when the gradient noise has finite variance.
Abstract:Adam retains a moving average of past squared gradients in its denominator, but the optimization cost of this memory is not well understood. We show that second-moment memory can itself suppress progress toward the optimum even under finite-variance stochastic gradients. For a simple two-point oracle, the expected positive normalized update is $O(M_2^{-1/2})$ after an initialization transient, where $M_2=(1-\beta_2)^{-1}$ is the second-moment memory length. We convert this directional bound, under the stated memory and stepsize scaling, into an average-stationarity lower bound of the same order on a smooth convex problem with normalized gap, smoothness, and variance. Long second-moment memory can slow optimization even when the gradient noise has finite variance.
Machine Learning (cs.LG)
arXiv:2608.15824 [cs.LG]
(or arXiv:2608.15824v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.15824 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 203 words
Related coverage
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
arXiv · Sep 1, 2026
PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent
arXiv · Oct 1, 2026
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
arXiv · Jul 21, 2026
Low-Rank Friction for Memory-Efficient Transformer Pretraining
arXiv · Sep 28, 2026