aiqbench analysis
This research introduces OptEMA, an adaptive optimizer that improves upon standard methods like Adam by achieving optimal convergence rates in both noisy and zero-noise regimes without requiring manual hyperparameter retuning. By utilizing a closed-loop design with trajectory-dependent stepsizes, the method eliminates reliance on prior knowledge of smoothness constants while maintaining Lipschitz-free properties for more robust training dynamics.
OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality
arXiv:2603.09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam. However, existing analyses of Adam-style methods often yield suboptimal guarantees in the zero-noise regime, rely on open-loop parameter schedules, or require prior knowledge of smoothness constants. Motivated by these limitations, we introduce OptEMA and analyze two complementary variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first moment with a fixed second-moment decay, and OptEMA-V, which swaps these roles. At the heart of these variants is a Corrected AdaGrad-Norm coefficient schedule. This formulation renders OptEMA algorithmically closed-loop and Lipschitz-free, meaning its effective stepsizes are trajectory-dependent and require no parameterization via the Lipschitz constant. Under lower-boundedness, unbiasedness, bounded variance, average smoothness, and a bounded stochastic-gradient condition used to control the adaptive normalizers, we prove that both variants achieve the unified noise-adaptive rate $\tilde{\mathcal{O}} \left(T^{-1/2}+\sigma^{1/2}T^{-1/4}\right)$ for the averaged gradient norm. In the zero-noise regime, these bounds automatically reduce to the nearly optimal deterministic rate $\widetilde{\mathcal{O}}(T^{-1/2})$ without manual hyperparameter retuning.
arXiv:2603.09923 (cs)
Abstract:Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam. However, existing analyses of Adam-style methods often yield suboptimal guarantees in the zero-noise regime, rely on open-loop parameter schedules, or require prior knowledge of smoothness constants. Motivated by these limitations, we introduce OptEMA and analyze two complementary variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first moment with a fixed second-moment decay, and OptEMA-V, which swaps these roles. At the heart of these variants is a Corrected AdaGrad-Norm coefficient schedule. This formulation renders OptEMA algorithmically closed-loop and Lipschitz-free, meaning its effective stepsizes are trajectory-dependent and require no parameterization via the Lipschitz constant. Under lower-boundedness, unbiasedness, bounded variance, average smoothness, and a bounded stochastic-gradient condition used to control the adaptive normalizers, we prove that both variants achieve the unified noise-adaptive rate $\tilde{\mathcal{O}} \left(T^{-1/2}+\sigma^{1/2}T^{-1/4}\right)$ for the averaged gradient norm. In the zero-noise regime, these bounds automatically reduce to the nearly optimal deterministic rate $\widetilde{\mathcal{O}}(T^{-1/2})$ without manual hyperparameter retuning.
Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC)
arXiv:2603.09923 [cs.LG]
(or arXiv:2603.09923v4 [cs.LG] for this version)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Achieving Exponential Asymptotic Optimality in Average-Reward Restless Bandits without Global Attractor Assumption
arXiv · Nov 3, 2026
Fundamental Limitations of QAOA on Constrained Problems and a Route to Exponential Enhancement
arXiv · Oct 15, 2026
Core-based Hierarchies for Efficient GraphRAG
arXiv · Jun 3, 2026
Near-Optimal Learning of Quantum Many-Body Systems from Gibbs and Thermal Metastable States via the Stationarity Test
arXiv · Nov 3, 2026