analiza aiqbench
PoLoRA introduces an optimizer specifically designed for Low-Rank Adaptation (LoRA) by leveraging spectral updates and curvature preconditioning, addressing the limitations of standard Adam in exploiting LoRA's matrix structure. Evaluation on code and math instruction-tuning tasks shows it converges 1.2 to 1.7 times faster than tuned Adam with minimal computational overhead while maintaining stability across different rank configurations. This advancement offers a more efficient training regime for fine-tuning large language models without sacrificing performance or requiring extensive hyperparameter tuning.
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
arXiv:2607.17620v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.
Abstract:Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.
Machine Learning (cs.LG); Computation and Language (cs.CL); Optimization and Control (math.OC)
arXiv:2607.17620 [cs.LG]
(or arXiv:2607.17620v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.17620 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron
NVIDIA · 22 apr. 2026
Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
arXiv · 21 sept. 2026
A Riemannian Geometry for Low-rank Adaptation
arXiv · 7 oct. 2026
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
arXiv · 30 sept. 2026