analiza aiqbench
This research introduces a specialized optimizer for Low-Rank Adaptation (LoRA) that leverages gauge symmetry to improve convergence rates and downstream performance in both supervised fine-tuning and reinforcement learning scenarios. By efficiently integrating rotation and normalization within the fixed-rank manifold, the method offers a significant advancement for efficient parameter-efficient fine-tuning strategies in machine learning workflows.
Rotated Manifold Optimization for Low-Rank Adaptation
arXiv:2609.34264v1 Announce Type: new Abstract: We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.
Abstract:We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.
Machine Learning (cs.LG)
arXiv:2609.34264 [cs.LG]
(or arXiv:2609.34264v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.34264 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 173 cuvinte
Subiecte conexe
Making LoRA Identifiable: Orthogonal Alignment for Continual Learning
arXiv · 14 oct. 2026
LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
arXiv · 3 sept. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent
arXiv · 9 sept. 2026