Adaptive Optimization via Momentum on Variance-Normalized Gradients

Abstract:We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization. MVN-Grad scales each coordinate by an exponential moving average of gradient uncertainty and applies momentum to the resulting normalized gradients, removing the cross-time coupling between stale momentum and a stochastic normalizer present in standard Adam-type updates. We prove that this decoupling yields smaller one-step conditional update variance than momentum-then-normalize variance methods, and that MVN-Grad has a uniformly bounded response to isolated gradient spikes. In low-variance regimes, we further show that variance normalization avoids sign-type collapse of second-moment scaling and can yield accelerated convergence. Beyond these comparisons, we prove a general nonconvex convergence guarantee for MVN-Grad under bounded-gradient stochastic assumptions. On CIFAR-100 and GPT-style language modeling, MVN-Grad matches or improves on Adam, AdaBelief, and LaProp, delivering smoother training and better generalization at the cost of one additional state tensor.

Machine Learning (cs.LG); Optimization and Control (math.OC)

arXiv:2602.10204 [cs.LG]

(or arXiv:2602.10204v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2602.10204 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.