analiza aiqbench
This paper presents a unified convergence theory for adaptive first-order optimization methods, rigorously analyzing popular algorithms like AdaGrad, AdaNorm, and Muon within nonconvex settings. By establishing global rate-of-convergence bounds without assuming bounded stochastic gradients or small stepsizes, the work provides a robust theoretical foundation for understanding how these techniques handle heterogeneous geometries across variable groups. The analysis covers both deterministic and fully stochastic scenarios, offering critical insights into the mathematical guarantees that underpin modern deep learning optimization practices.
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muon
arXiv:2604.17423v3 Announce Type: replace Abstract: A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as an adpative variant of Muon. This framework also allows combining heterogeneous geometries across different groups of variables while preserving a unified convergence analysis. A fully stochastic global rate-of-convergence analysis is conducted for all methods in the framework, with andwithout two types of momentum, using reasonable assumptions on the variance of the gradient oracle and without assuming bounded stochastic gradients or small enough stepsize.
Abstract:A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as an adpative variant of Muon. This framework also allows combining heterogeneous geometries across different groups of variables while preserving a unified convergence analysis. A fully stochastic global rate-of-convergence analysis is conducted for all methods in the framework, with andwithout two types of momentum, using reasonable assumptions on the variance of the gradient oracle and without assuming bounded stochastic gradients or small enough stepsize.
Machine Learning (cs.LG)
I.2.6; F.2.1; G.1.6
arXiv:2604.17423 [cs.LG]
(or arXiv:2604.17423v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.17423 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 187 cuvinte
