aiqbench analysis
This paper proposes AdaSub-ZO, a forward-only way to fine-tune LLMs that learns low-rank projection matrices from finite-difference feedback instead of using fixed random subspaces. It targets the high gradient-estimation variance that limits zeroth-order methods, which matter for memory-constrained fine-tuning. The results are reported only on natural language understanding benchmarks, so readers should check how the gains hold on larger models and tasks before treating them as general.
Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
arXiv:2610.15225v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance in high-dimensional parameter spaces. While recent methods restrict perturbations to random low-rank subspaces, these subspaces often misalign with task-specific descent directions. We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces. Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions. Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases. Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.
Abstract:Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance in high-dimensional parameter spaces. While recent methods restrict perturbations to random low-rank subspaces, these subspaces often misalign with task-specific descent directions. We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces. Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions. Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases. Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.
Machine Learning (cs.LG)
arXiv:2610.15225 [cs.LG]
(or arXiv:2610.15225v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.15225 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 206 words
Related coverage
StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning
arXiv · Oct 15, 2026
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
arXiv · Sep 30, 2026
SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
arXiv · Aug 19, 2026
Low Kruskal-Rank Adaptation
AMD · Jun 11, 2026