analiza aiqbench
This arXiv paper proposes SaMA, a parameter-efficient fine-tuning method for large language models that generalizes the Kronecker product to lift the rank limits of LoRA. The authors claim its effective rank grows quadratically rather than linearly with parameter budget, and that its invariance structure is easier for adaptive optimizers to handle. These are claims from a preprint with empirical results on commonsense and arithmetic reasoning benchmarks, so readers should check them against independent replications and the cost-accuracy comparisons with other PEFT baselines before treating SaMA as a practical replacement for LoRA.
SaMA: Morpho Adaptation via Asymmetric Expansion of Kronecker Product
arXiv:2611.04462v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has attracted significant attention in the parameter-efficient fine-tuning (PEFT) landscape. However, its design suffers from two fundamental limitations. First, its invariance manifold induces flat directions in the loss landscape, creating an optimization bottleneck for adaptive optimizers. Second, its effective rank scales only linearly with the parameter budget. In this paper, we propose Scalable Morpho Adaptation (SaMA), a PEFT method grounded in a principled generalization of the Kronecker product. By relaxing the shared-block constraint in the perfect-shuffle factorization, SaMA spans from low to full rank and achieves a practical rank that scales quadratically with the parameter budget, up to full-rank. Furthermore, we show that its invariance manifold collapses to a diagonal group, making it structurally more optimizer-friendly. Empirically, SaMA consistently outperforms strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while being more parameter-efficient.
Abstract:Low-rank adaptation (LoRA) has attracted significant attention in the parameter-efficient fine-tuning (PEFT) landscape. However, its design suffers from two fundamental limitations. First, its invariance manifold induces flat directions in the loss landscape, creating an optimization bottleneck for adaptive optimizers. Second, its effective rank scales only linearly with the parameter budget. In this paper, we propose Scalable Morpho Adaptation (SaMA), a PEFT method grounded in a principled generalization of the Kronecker product. By relaxing the shared-block constraint in the perfect-shuffle factorization, SaMA spans from low to full rank and achieves a practical rank that scales quadratically with the parameter budget, up to full-rank. Furthermore, we show that its invariance manifold collapses to a diagonal group, making it structurally more optimizer-friendly. Empirically, SaMA consistently outperforms strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while being more parameter-efficient.
Accepted at NeurIPS 2026. 61 pages, 29 figures, 25 tables. Parameter-Efficient Fine-Tuning (PEFT); Large Language Models (LLMs); Large Foundation Models
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2611.04462 [cs.LG]
(or arXiv:2611.04462v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.04462 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 177 cuvinte
Subiecte conexe
A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks
arXiv · 12 iun. 2026
Anchor and Adapt: Asymmetric Prompt Adaptation for Few-Shot Industrial Anomaly Detection
arXiv · 7 oct. 2026
Align Then Adapt: Label-Efficient Adapter Learning for Asymmetric Dense Retrieval
arXiv · 27 aug. 2026
Receiver-Centered Robot-to-Human Handover with Grasp-Aware Object Orientation
arXiv · 21 iul. 2026