aiqbench analysis
This research addresses a critical inefficiency in Low-Rank Adaptation (LoRA) by introducing adaptive anisotropic learning rates that account for heterogeneity within adapter modules. By dynamically assigning individual learning rates based on training signals like function-space velocity and Adam SNR, the proposed method maximizes rank capacity utilization without adding trainable parameters. The approach demonstrates robust performance gains across diverse benchmarks, offering a practical optimization for parameter-efficient fine-tuning workflows.
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
arXiv:2609.05885v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.
Abstract:Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.
Machine Learning (cs.LG)
arXiv:2609.05885 [cs.LG]
(or arXiv:2609.05885v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.05885 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 267 words
Related coverage
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
arXiv · Jun 16, 2026
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
LessWrong · Oct 2, 2026
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face · Sep 10, 2026