aiqbench analysis
This research introduces a novel stochastic doubly smoothed gradient method designed to tackle nonconvex federated stochastic bilevel optimization problems without relying on restrictive assumptions about lower-level convexity. By decoupling learning rates for upper- and lower-level variables, the algorithm achieves improved convergence rates with significantly reduced dependence on the condition number, addressing key bottlenecks in current first-order methods. Extensive experiments confirm the approach's effectiveness, offering a more practical solution for hyperparameter tuning and communication efficiency in distributed machine learning systems.
Smoothed Gradient Method for Nonconvex Federated Stochastic Bilevel Optimization
arXiv:2610.05290v1 Announce Type: new Abstract: In recent years, federated stochastic bilevel optimization has attracted increasing attention due to its wide range of applications in machine learning. To reduce the computational overhead associated with second-order Hessian and Jacobian matrices, several first-order methods have been proposed. However, existing methods typically impose restrictive assumptions on the lower-level function, suffer from a strong dependence on the condition number in their convergence rates, and require different learning-rate scales for variables across the upper- and lower-level problems, limiting their practical applicability and complicating hyperparameter tuning. To address these challenges, we propose a stochastic doubly smoothed gradient method for nonconvex federated stochastic bilevel optimization problems, which decouples the learning rates of upper- and lower-level variables and does not require a strongly-convex lower-level loss function. We establish rigorous theoretical guarantees for the proposed algorithm, demonstrating an improved convergence rate of $O(\kappa^{15/2}/\epsilon^5)$ and a communication complexity of $O(\kappa^{4}/\epsilon^3)$, where $\kappa$ denotes the condition number and $\epsilon$ represents the solution accuracy. Notably, these bounds exhibit significantly better dependence on the condition number $\kappa$ than those of existing methods. Extensive experiments validate the effectiveness of our algorithm.
Abstract:In recent years, federated stochastic bilevel optimization has attracted increasing attention due to its wide range of applications in machine learning. To reduce the computational overhead associated with second-order Hessian and Jacobian matrices, several first-order methods have been proposed. However, existing methods typically impose restrictive assumptions on the lower-level function, suffer from a strong dependence on the condition number in their convergence rates, and require different learning-rate scales for variables across the upper- and lower-level problems, limiting their practical applicability and complicating hyperparameter tuning. To address these challenges, we propose a stochastic doubly smoothed gradient method for nonconvex federated stochastic bilevel optimization problems, which decouples the learning rates of upper- and lower-level variables and does not require a strongly-convex lower-level loss function. We establish rigorous theoretical guarantees for the proposed algorithm, demonstrating an improved convergence rate of $O(\kappa^{15/2}/\epsilon^5)$ and a communication complexity of $O(\kappa^{4}/\epsilon^3)$, where $\kappa$ denotes the condition number and $\epsilon$ represents the solution accuracy. Notably, these bounds exhibit significantly better dependence on the condition number $\kappa$ than those of existing methods. Extensive experiments validate the effectiveness of our algorithm.
Machine Learning (cs.LG)
arXiv:2610.05290 [cs.LG]
(or arXiv:2610.05290v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.05290 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 275 words
Related coverage
A Single-Loop, Constant-Batch First-Order Penalty Method for Stochastic Bilevel Optimization
arXiv · Oct 7, 2026
SGHA: A Single-Loop Fully First-Order Algorithm for Nonconvex-Strongly-Convex Bilevel Optimization
arXiv · Aug 25, 2026
Optimal Stochastic Bilevel Optimization with First-Order Oracles
arXiv · Oct 2, 2026
Stochastic Bilevel Optimization with Heavy-Tailed Noise
arXiv · Sep 28, 2026