analiza aiqbench
SubZero+ addresses critical stability issues in zeroth-order optimization by introducing multi-query gradient estimation and adaptive subspace optimizers, enabling reliable backpropagation-free fine-tuning for large language models without increasing memory overhead. This advancement significantly expands the stable learning-rate range and narrows performance gaps compared to traditional first-order methods across various model sizes.
SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
arXiv:2608.15665v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates. We propose SubZero+, an improved SubZero framework that improves stability in three complementary ways: (i) multi-query gradient estimation within layer-specific low-rank subspaces to reduce variance without exhibiting the multi-query paradox; (ii) a subspace Adam optimizer that performs adaptive updates using in-subspace multi-query gradient statistics; and (iii) a sign correction for QR-based subspace construction to ensure Haar-distributed projection matrices, eliminating implementation-dependent orientation ambiguity. Experiments on models from 1.3B to 32B across SuperGLUE, under both full-parameter tuning and LoRA, show that SubZero+ consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.
Abstract:Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates. We propose SubZero+, an improved SubZero framework that improves stability in three complementary ways: (i) multi-query gradient estimation within layer-specific low-rank subspaces to reduce variance without exhibiting the multi-query paradox; (ii) a subspace Adam optimizer that performs adaptive updates using in-subspace multi-query gradient statistics; and (iii) a sign correction for QR-based subspace construction to ensure Haar-distributed projection matrices, eliminating implementation-dependent orientation ambiguity. Experiments on models from 1.3B to 32B across SuperGLUE, under both full-parameter tuning and LoRA, show that SubZero+ consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.
Machine Learning (cs.LG)
arXiv:2608.15665 [cs.LG]
(or arXiv:2608.15665v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.15665 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 224 cuvinte
Subiecte conexe
StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning
arXiv · 15 oct. 2026
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
arXiv · 16 iun. 2026
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
arXiv · 3 iun. 2026
SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs
arXiv · 2 oct. 2026