aiqbench analysis
This arXiv paper gives theoretical convergence guarantees for clipped stochastic gradient descent when gradient noise is heavy-tailed, with bounds for the final iterate rather than an averaged one. The results matter to practitioners training models on noisy or unbounded gradients, since they quantify how clipping and a linearly decaying learning rate affect error rates, and how a limited number of adversarially corrupted gradients degrades them. The work is purely theoretical, so its practical value depends on whether the assumptions (unbiased gradients, bounded central p-th moment) hold in real training pipelines.
Last-Iterate Convergence of Clipped SGD with Linear Decay
arXiv:2610.18555v1 Announce Type: new Abstract: In this paper, we study the last-iterate convergence of clipped stochastic gradient descent under heavy-tailed noise. We assume that the stochastic gradients are unbiased and that the noise has a bounded central $p$-th moment, where $1<p\le2$. For smooth convex objectives, a linearly decreasing learning rate gives a high-probability error bound of order $O(LR^2\log(1/\delta)/T+\sigma R(\log(1/\delta)/T)^{(p-1)/p})$. We further establish last-iterate guarantees for nonsmooth composite objectives and smooth composite objectives with proximal updates. In addition, when a causal adversary replaces at most $q$ gradients, the same clipped update achieves a bound with $\log(1/\delta)$ replaced by $\log(1/\delta)+q$. The corresponding lower bounds for the clean and corrupted oracle models are also provided.
Abstract:In this paper, we study the last-iterate convergence of clipped stochastic gradient descent under heavy-tailed noise. We assume that the stochastic gradients are unbiased and that the noise has a bounded central $p$-th moment, where $1<p\le2$. For smooth convex objectives, a linearly decreasing learning rate gives a high-probability error bound of order $O(LR^2\log(1/\delta)/T+\sigma R(\log(1/\delta)/T)^{(p-1)/p})$. We further establish last-iterate guarantees for nonsmooth composite objectives and smooth composite objectives with proximal updates. In addition, when a causal adversary replaces at most $q$ gradients, the same clipped update achieves a bound with $\log(1/\delta)$ replaced by $\log(1/\delta)+q$. The corresponding lower bounds for the clean and corrupted oracle models are also provided.
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2610.18555 [cs.LG]
(or arXiv:2610.18555v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.18555 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 205 words
Related coverage
On the Last-Iterate Convergence of Clipped Gradient Methods
arXiv · Oct 16, 2026
Koopman Theory and Metaethics
LessWrong · Sep 19, 2026
Pre-Training Hybrid LLMs from Scratch in Primus on AMD Instinct™ GPUs
AMD · Oct 8, 2026
Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
arXiv · Aug 21, 2026