analiza aiqbench
The paper proposes a differentiable version of stopping time, the time needed to reach a target loss, which has usually been treated as non-differentiable and optimized with slower zeroth-order methods. If the approach holds up, it would let gradient-based tools tune hyperparameters for faster convergence, a practical gain for training and online tuning. Readers should note that the claims rest on the authors' experiments across selected problems, so the method's advantage over existing baselines needs independent testing.
Accelerating Optimization via Differentiable Stopping Time
arXiv:2505.22509v2 Announce Type: replace Abstract: Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.
Abstract:Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.
Corrected bibliographic metadata. No changes to the main text or appendix
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2505.22509 [cs.LG]
(or arXiv:2505.22509v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2505.22509 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 261 cuvinte
Subiecte conexe
Differential acceleration of alignment-relevant capabilities is a bad bet
LessWrong · 21 iul. 2026
A basic case for differentially accelerating conceptual reasoning capabilities
LessWrong · 26 oct. 2026
Fast, accurate, and differentiable: a neural-network surrogate for NRSur7dq4 precessing binary black hole waveforms
arXiv · 29 iul. 2026
CommentBench: Can Models Match Human Comments on AI Safety Posts?
LessWrong · 19 sept. 2026