aiqbench analysis
This research establishes theoretical scaling laws for SignSGD within linear regression, demonstrating how its unique noise-reshaping effects can yield steeper compute-optimal slopes than standard SGD in high-noise regimes. The study further validates that the warmup-stable-decay schedule effectively mitigates noise terms when feature decay is rapid relative to target decay. These findings provide critical insights for optimizing training efficiency and model convergence in large-scale linear models.
Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
arXiv:2603.02069v2 Announce Type: replace Abstract: We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.
Title:Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
Abstract:We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
arXiv:2603.02069 [cs.LG]
(or arXiv:2603.02069v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.02069 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
arXiv · Sep 11, 2026
Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling
arXiv · Jul 23, 2026
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
arXiv · Sep 9, 2026
Train-to-Test (T²) Scaling Laws: Why Reasoning Models Should Be Overtrained
Snorkel AI Blog · Aug 18, 2026