analiza aiqbench
LAARA introduces a search-free method for dynamically allocating adapter ranks in parameter-efficient fine-tuning, addressing the inefficiency of uniform rank assignments across transformer layers. By leveraging lightweight diagonal Fisher estimates during training, this framework achieves performance comparable to or better than state-of-the-art approaches like LoRA while significantly reducing trainable parameters. The technique offers a principled optimization strategy that could enhance model efficiency and adaptability in large-scale machine learning workflows.
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
arXiv:2607.19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: https://anonymous.4open.science/r/LAARA-D305/LAARA.py
Abstract:Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.19391 [cs.LG]
(or arXiv:2607.19391v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.19391 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 246 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 10 sept. 2026
7 Approaches to Efficient LLM Training on Limited Hardware
KDnuggets · 9 sept. 2026
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
arXiv · 5 aug. 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · 11 mai 2026