analiza aiqbench
VRAdam introduces a physics-inspired velocity regularizer that dynamically dampens oscillations in high-velocity regimes, offering a theoretically grounded alternative to standard optimizers like AdamW. By combining global damping with per-parameter scaling, the method achieves improved convergence rates and performance across diverse tasks including image classification and language modeling. This approach provides a rigorous control-theoretic perspective on training dynamics, potentially addressing the stability limitations inherent in current adaptive edge-of-stability algorithms.
A Physics-Inspired Optimizer: Velocity Regularized Adam
arXiv:2505.13196v3 Announce Type: replace Abstract: We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics. Previous algorithms, including the ubiquitous Adam, operate at the so-called adaptive edge of stability regime during training, leading to rapid oscillations and slowed convergence of loss. However, VRAdam adds a higher order penalty on the learning rate based on the velocity such that the algorithm automatically slows down whenever weight updates become large. In practice, we observe that the effective dynamic learning rate shrinks in high-velocity regimes, and damping oscillations. By combining this velocity-based regularizer for global damping with per-parameter scaling of Adam, we create a powerful hybrid optimizer. For this optimizer, we provide rigorous theoretical analysis of operation at the edge of stability from a physical and control perspective for the momentum. Furthermore, we derive convergence bounds with the rate $\mathcal{O}(\ln(N)/\sqrt{N})$ for a stochastic non convex objective under mild assumptions. We demonstrate that VRAdam exceeds the performance against standard optimizers including AdamW. We benchmark various tasks such as image classification, language modeling, and generative modeling using diverse architectures and training methodologies including Convolutional Neural Networks (CNNs), Transformers, and GFlowNets.
Abstract:We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics. Previous algorithms, including the ubiquitous Adam, operate at the so-called adaptive edge of stability regime during training, leading to rapid oscillations and slowed convergence of loss. However, VRAdam adds a higher order penalty on the learning rate based on the velocity such that the algorithm automatically slows down whenever weight updates become large. In practice, we observe that the effective dynamic learning rate shrinks in high-velocity regimes, and damping oscillations. By combining this velocity-based regularizer for global damping with per-parameter scaling of Adam, we create a powerful hybrid optimizer. For this optimizer, we provide rigorous theoretical analysis of operation at the edge of stability from a physical and control perspective for the momentum. Furthermore, we derive convergence bounds with the rate $\mathcal{O}(\ln(N)/\sqrt{N})$ for a stochastic non convex objective under mild assumptions. We demonstrate that VRAdam exceeds the performance against standard optimizers including AdamW. We benchmark various tasks such as image classification, language modeling, and generative modeling using diverse architectures and training methodologies including Convolutional Neural Networks (CNNs), Transformers, and GFlowNets.
L. Schorling and P. Vaidhyanathan contributed equally to this work. 20 pages, 10 figures
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Quantum Physics (quant-ph)
arXiv:2505.13196 [cs.LG]
(or arXiv:2505.13196v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 232 cuvinte
Subiecte conexe
Generalization and infinite width
LessWrong · 1 aug. 2026
Tariffs didn’t bring manufacturing jobs back to the US
The Verge · 27 iul. 2026
A brief history of quantum vs classical computational advantage
Quantum Journal · 1 sept. 2026
Researchers Stabilise Neural Networks for Quantum Simulations
Quantum Zeitgeist · 14 sept. 2026