aiqbench analysis
This research provides a mechanistic explanation for why setting the momentum parameters equal ($\beta_1 = \beta_2$) in Adam optimizers yields superior stability, identifying the vanishing of a specific "magnitude-lag" term as the key dynamic factor. By decomposing updates into sign and lag components, the study reveals that tied configurations eliminate mismatch-induced responses, resulting in smoother training trajectories across vision and language tasks. These findings offer critical theoretical insight for optimizing large-scale model training by clarifying the structural role of memory scale alignment.
Why $\beta_1 = \beta_2$ Is Dynamically Special in Adam
arXiv:2601.21739v3 Announce Type: replace-cross Abstract: Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying $\beta_{1}=\beta_{2}$ can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when $\beta_{1}=\beta_{2}$, making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.
Abstract:Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying $\beta_{1}=\beta_{2}$ can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when $\beta_{1}=\beta_{2}$, making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.
28 pages, 8 figures. Preprint
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2601.21739 [cs.LG]
(or arXiv:2601.21739v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.21739 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
Provable Edge-of-Stability for Adam on a One-Dimensional Quadratic
arXiv · Aug 24, 2026
Adam Conover explains how YouTube ruined everything
The Verge · Sep 14, 2026
Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer
arXiv · Sep 30, 2026
Beyond Quadratic Loss: The Stability Phase Diagram of Adam
arXiv · Sep 17, 2026