aiqbench analysis
This arXiv paper is about optimization theory for training ML models, specifically asynchronous distributed training under heavy-tailed gradient noise, which is common in transformers. The authors propose delay-aware learning-rate scheduling and delay compensation, and claim optimal convergence rates plus better tolerance to stragglers. Their empirical results cover image and language tasks, but they are preprint results, so practitioners should check them against their own distributed setups before relying on them.
Asynchronous Heavy-Tailed Optimization
arXiv:2602.18002v2 Announce Type: replace Abstract: Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, while asynchronous optimization under such noise remains underexplored, especially in the practical settings with local updates. In this work, we first investigate two asynchronous communication schemes (server- and client-centric) that handle stragglers under heavy-tailed gradient noise. We further propose and theoretically analyze algorithmic modifications based on delay-aware learning rate scheduling and delay compensation to enhance the performance of asynchronous algorithms. Our convergence guarantees achieve the optimal rate under heavy-tailed noise and improve delay tolerance compared with existing asynchronous approaches. Empirically, our approaches outperform prior synchronous and asynchronous methods in terms of accuracy/runtime trade-offs and are more robust to hyperparameters in both image and language tasks.
Abstract:Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, while asynchronous optimization under such noise remains underexplored, especially in the practical settings with local updates. In this work, we first investigate two asynchronous communication schemes (server- and client-centric) that handle stragglers under heavy-tailed gradient noise. We further propose and theoretically analyze algorithmic modifications based on delay-aware learning rate scheduling and delay compensation to enhance the performance of asynchronous algorithms. Our convergence guarantees achieve the optimal rate under heavy-tailed noise and improve delay tolerance compared with existing asynchronous approaches. Empirically, our approaches outperform prior synchronous and asynchronous methods in terms of accuracy/runtime trade-offs and are more robust to hyperparameters in both image and language tasks.
Machine Learning (cs.LG)
arXiv:2602.18002 [cs.LG]
(or arXiv:2602.18002v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.18002 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 231 words
Related coverage
Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
arXiv · Nov 2, 2026
SmoothRL: Online Reinforcement Learning During Asynchronous Execution
arXiv · Sep 2, 2026
High dimensional theory of two-phase optimizers
arXiv · Oct 6, 2026
Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods
arXiv · Oct 20, 2026