aiqbench analysis
This research advances federated learning by introducing a first-order gradient algorithm that eliminates the computationally expensive second-order matrix calculations typical of bilevel optimization. By utilizing a constant single-timescale learning rate, the method significantly reduces training time while maintaining convergence guarantees for distributed machine learning tasks.
Federated stochastic bilevel optimization with fully first-order gradients
arXiv:2609.16350v1 Announce Type: new Abstract: Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, most existing federated stochastic bilevel optimization algorithms require the computation of second-order Hessian and Jacobian matrices, which leads to longer running times in practice. To address these challenges, we propose a novel federated stochastic variance-reduced bilevel gradient descent algorithm that relies solely on first-order oracles. Specifically, our approach does not require the computation of second-order Hessian and Jacobian matrices, significantly reducing running time. Furthermore, we introduce a novel learning rate mechanism, i.e., a constant single-timescale learning rate, to coordinate the update of different variables. We also present a new strategy to establish the convergence rate of our algorithm. Finally, the extensive experimental results confirm the efficacy of our proposed algorithm.
Abstract:Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, most existing federated stochastic bilevel optimization algorithms require the computation of second-order Hessian and Jacobian matrices, which leads to longer running times in practice. To address these challenges, we propose a novel federated stochastic variance-reduced bilevel gradient descent algorithm that relies solely on first-order oracles. Specifically, our approach does not require the computation of second-order Hessian and Jacobian matrices, significantly reducing running time. Furthermore, we introduce a novel learning rate mechanism, i.e., a constant single-timescale learning rate, to coordinate the update of different variables. We also present a new strategy to establish the convergence rate of our algorithm. Finally, the extensive experimental results confirm the efficacy of our proposed algorithm.
Accepted for publication in the Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2025). The official version is available at this https URL
Machine Learning (cs.LG)
arXiv:2609.16350 [cs.LG]
(or arXiv:2609.16350v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.16350 arXiv-issued DOI via DataCite (pending registration)
In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2025) (pp. 7047-7055)
https://doi.org/10.24963/ijcai.2025/784 DOI(s) linking to related resources
Full text under the publisher's open licence · 195 words
Related coverage
A Single-Loop, Constant-Batch First-Order Penalty Method for Stochastic Bilevel Optimization
arXiv · Oct 7, 2026
Smoothed Gradient Method for Nonconvex Federated Stochastic Bilevel Optimization
arXiv · Oct 6, 2026
SGHA: A Single-Loop Fully First-Order Algorithm for Nonconvex-Strongly-Convex Bilevel Optimization
arXiv · Aug 25, 2026
Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles
arXiv · Jun 15, 2026