aiqbench analysis
This study provides a rigorous empirical separation of spectral band selection from input/output factor freezing in Low-Rank Adaptation (LoRA), demonstrating that freezing the input factor A yields consistent performance gains over freezing B or random bases at low ranks. The findings clarify how these architectural choices interact with rank and spectral placement, offering actionable guidance for optimizing parameter-efficient fine-tuning without relying on inflated significance claims.
Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA
arXiv:2610.06621v1 Announce Type: cross Abstract: Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.
Title:Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA
Abstract:Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.06621 [cs.LG]
(or arXiv:2610.06621v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.06621 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
random_A_results.json
supplementary_record.pdf
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 320 words
Related coverage
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
arXiv · Jun 12, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
arXiv · Sep 2, 2026