analiza aiqbench
This paper introduces Manifold-LoRA, an optimization method for fine-tuning large language models that eliminates the need for costly retraction operators or penalty parameter tuning by leveraging Stiefel manifold geometry. By reformulating LoRA as a direct landing problem on this manifold, the approach achieves accelerated training with global convergence guarantees comparable to existing state-of-the-art complexities. The technique offers a theoretically grounded path toward more efficient adaptation of foundation models without sacrificing downstream performance.
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
arXiv:2607.25299v1 Announce Type: cross Abstract: Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
Abstract:Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.25299 [cs.LG]
(or arXiv:2607.25299v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.25299 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
arXiv · 21 sept. 2026
LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
arXiv · 3 sept. 2026
Making LoRA Identifiable: Orthogonal Alignment for Continual Learning
arXiv · 14 oct. 2026
Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
arXiv · 18 sept. 2026