analiza aiqbench
This research challenges the conventional wisdom that gradient clipping universally enhances statistical accuracy in averaged SGD, particularly when dealing with heavy-tailed noise distributions. The authors demonstrate through finite-sample analysis that for moments $p>2$, the Gaussian component often dominates deviation bounds, meaning clipping may not improve leading-order performance and can even increase asymptotic variance or shift limiting points under asymmetric conditions. These findings provide critical theoretical guidance for practitioners deciding whether to apply clipping thresholds in robust optimization scenarios.
Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise
arXiv:2609.32962v1 Announce Type: new Abstract: Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional $p$-th moments, $p\ge2$. Our main result gives explicit accuracy and confidence conditions under which, for $p>2$, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At $p=2$, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
Title:Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise
Abstract:Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional $p$-th moments, $p\ge2$. Our main result gives explicit accuracy and confidence conditions under which, for $p>2$, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At $p=2$, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2609.32962 [cs.LG]
(or arXiv:2609.32962v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.32962 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
arXiv · 24 sept. 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · 15 oct. 2026
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · 21 iul. 2026
Protocol balances accuracy and privacy in quantum sensing networks
Quantum Zeitgeist · 23 aug. 2026