analiza aiqbench
RoSA introduces a novel layer-freezing strategy that progressively rotates trainable blocks to optimize memory usage during large language model fine-tuning. By reducing optimizer-state requirements and shortening backpropagation paths, this method offers a practical approach to scaling parameter-efficient fine-tuning without sacrificing performance. The technique's orthogonality to existing PEFT methods suggests broad applicability across various foundation model architectures.
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
arXiv:2610.06243v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.
Abstract:Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.
Accepted at NeurIPS 2026. 16 pages, 5 figures
Machine Learning (cs.LG)
arXiv:2610.06243 [cs.LG]
(or arXiv:2610.06243v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.06243 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 232 cuvinte
Subiecte conexe
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
arXiv · 15 iun. 2026
Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
arXiv · 3 iun. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026