aiqbench analysis
LOFT introduces a unified mathematical framework for orthogonal parameter-efficient fine-tuning (PEFT) by explicitly decoupling subspace rotation from transformation mechanics. By integrating task-aware support selection strategies, the method demonstrates improved performance across diverse benchmarks like language understanding and mathematical reasoning while maintaining strict parameter budgets. This work advances the theoretical understanding of how to optimize low-rank adaptations without conflating distinct design choices in PEFT architectures.
LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
From arXiv
Full textarXiv:2605.11872v2 Announce Type: replace Abstract: Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers the multiplicative core of representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder- (with a learnable support) and principal-subspace-based variants (up to scaling and relaxation).
More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT with task-aware supports improves performance at matched trainable-parameter and transform settings under a shared protocol. This suggests that, under a fixed transform and budget, the choice of support affects performance, making systematic support selection an important direction for improving orthogonal PEFT.
Abstract:Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers the multiplicative core of representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder- (with a learnable support) and principal-subspace-based variants (up to scaling and relaxation).
More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT with task-aware supports improves performance at matched trainable-parameter and transform settings under a shared protocol. This suggests that, under a fixed transform and budget, the choice of support affects performance, making systematic support selection an important direction for improving orthogonal PEFT.
Accepted at NeurIPS 2026. Camera-ready version
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2605.11872 [cs.LG]
(or arXiv:2605.11872v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.11872 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words
Related coverage
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
arXiv · Aug 20, 2026
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
arXiv · Jun 16, 2026
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
arXiv · Jul 21, 2026
CORAM: Coherent Orthogonal Rotation for Model Merging
arXiv · Aug 20, 2026