analiza aiqbench
This research introduces NoRA, a normalization technique for Low-Rank Adaptation (LoRA) that stabilizes training dynamics by addressing the asymmetry in initialization between up- and down-projection matrices. By applying this simple adjustment either during training or solely at initialization, the method accelerates convergence and improves performance across pretraining, supervised finetuning, and reinforcement learning without adding parameters or inference overhead. It represents a significant optimization for parameter-efficient fine-tuning workflows where training stability is critical.
Normalized Low-Rank Adaptation
arXiv:2608.31036v1 Announce Type: new Abstract: While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.
Abstract:While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.
Machine Learning (cs.LG)
arXiv:2608.31036 [cs.LG]
(or arXiv:2608.31036v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.31036 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 219 cuvinte
Subiecte conexe
Evaluating task vectors, unlearning and inoculation
LessWrong · 20 sept. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face · 10 sept. 2026
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
arXiv · 30 sept. 2026