aiqbench analysis
PowerStep introduces a novel optimization algorithm that significantly reduces memory overhead during Transformer training by eliminating the need to store second-moment statistics. By leveraging $\ell_p$-norm steepest descent, it achieves competitive performance on models ranging from 124M to 235B parameters while cutting optimizer-state memory usage by up to eight times when combined with quantization. This advancement addresses a critical bottleneck in scaling large AI models without compromising numerical stability or validation quality.
PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent
arXiv:2605.10335v2 Announce Type: replace-cross Abstract: Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by $\ell_p$-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an $O(1/\sqrt{T})$ term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving $\texttt{fp32}$ optimizer-state memory relative to AdamW. Combined with uniform $\texttt{int8}$ quantization, PowerStep remains numerically stable and reduces optimizer-state memory by $\sim8\times$ compared to $\texttt{fp32}$ AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.
Abstract:Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by $\ell_p$-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an $O(1/\sqrt{T})$ term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving $\texttt{fp32}$ optimizer-state memory relative to AdamW. Combined with uniform $\texttt{int8}$ quantization, PowerStep remains numerically stable and reduces optimizer-state memory by $\sim8\times$ compared to $\texttt{fp32}$ AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Numerical Analysis (math.NA); Optimization and Control (math.OC)
arXiv:2605.10335 [cs.LG]
(or arXiv:2605.10335v2 [cs.LG] for this version)
Full text under the publisher's open licence · 142 words
Related coverage
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · Jul 23, 2026
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
arXiv · Oct 2, 2026
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
arXiv · Jun 12, 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026