aiqbench analysis
This paper provides a rigorous Lyapunov-based analysis of Stochastic Gradient Descent with Momentum under dynamic learning-rate and batch-size schedules. The study establishes that increasing both hyperparameters simultaneously yields provably faster convergence than fixed strategies, offering theoretical justification for common deep learning tuning practices.
Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis
arXiv:2508.03105v3 Announce Type: replace Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm under our Lyapunov-based analysis, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warmup schedule in experiments, which empirically outperformed all other strategies in convergence behavior.
Abstract:We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm under our Lyapunov-based analysis, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warmup schedule in experiments, which empirically outperformed all other strategies in convergence behavior.
Accepted at TMLR (Jul. 2026)
Machine Learning (cs.LG)
arXiv:2508.03105 [cs.LG]
(or arXiv:2508.03105v3 [cs.LG] for this version)
Full text under the publisher's open licence · 168 words
Related coverage
Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
arXiv · Jul 28, 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · May 11, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026
[AINews] AI Engineer NYC: Gemini 4 Argon, Vals AI Benchmarks
Latent Space · Oct 14, 2026