analiza aiqbench
This paper introduces RanSOM, a novel optimization algorithm that leverages randomized step lengths to eliminate the need for auxiliary points or Lipschitz Hessian assumptions in second-order momentum methods. By utilizing Stein-type identities, the method achieves optimal oracle-complexity exponents even under heavy-tailed noise while maintaining computational efficiency comparable to existing approaches on standard benchmarks like CIFAR-10. The work represents a significant theoretical and practical advancement for training deep neural networks where gradient noise characteristics vary.
RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
arXiv:2602.06824v3 Announce Type: replace-cross Abstract: Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extra forward and backward passes at an auxiliary point. We show that randomizing the step length removes this trade-off: if the step is exponentially distributed with mean $\eta$, a Stein-type identity makes $\eta$ times one HVP at the new iterate an estimate of the gradient change that is unbiased in expectation over the step, and this HVP shares its forward pass with the next gradient. The resulting method, RanSOM-E, works with normalized, sign, and Muon-type steps, and a Beta-distributed variant, RanSOM-B, keeps Frank-Wolfe iterates feasible. For objectives with Lipschitz gradient and heavy-tailed gradient and Hessian noise with tail indices $p, q \in (1,2]$, RanSOM-E attains the optimal oracle-complexity exponents for every pair $(p,q)$, including $p \neq q$, without a Lipschitz Hessian or clipping ($\mathcal{O}(\varepsilon^{-3})$ for $p = q = 2$). On CIFAR-10 with ResNet-20, we detect no accuracy difference from existing second-order momentum methods, and its mean update time is lower than that of the auxiliary-point method.
Abstract:Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extra forward and backward passes at an auxiliary point. We show that randomizing the step length removes this trade-off: if the step is exponentially distributed with mean $\eta$, a Stein-type identity makes $\eta$ times one HVP at the new iterate an estimate of the gradient change that is unbiased in expectation over the step, and this HVP shares its forward pass with the next gradient. The resulting method, RanSOM-E, works with normalized, sign, and Muon-type steps, and a Beta-distributed variant, RanSOM-B, keeps Frank-Wolfe iterates feasible. For objectives with Lipschitz gradient and heavy-tailed gradient and Hessian noise with tail indices $p, q \in (1,2]$, RanSOM-E attains the optimal oracle-complexity exponents for every pair $(p,q)$, including $p \neq q$, without a Lipschitz Hessian or clipping ($\mathcal{O}(\varepsilon^{-3})$ for $p = q = 2$). On CIFAR-10 with ResNet-20, we detect no accuracy difference from existing second-order momentum methods, and its mean update time is lower than that of the auxiliary-point method.
Optimization and Control (math.OC); Machine Learning (cs.LG)
arXiv:2602.06824 [math.OC]
(or arXiv:2602.06824v3 [math.OC] for this version)
https://doi.org/10.48550/arXiv.2602.06824 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 278 cuvinte
