UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Abstract:Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allocation across weight matrices: uniform or hand-crafted rules ignore module-wise importance, while learning-based allocation incurs substantial training overhead. We formulate rank allocation as a global sorting-and-truncation pipeline that scores every singular component by combining local singular energy with global functional importance, estimated via layer-wise input--output cosine similarity on a tiny calibration set. We show, both geometrically and empirically, that high input--output cosine similarity implies low effective rank. We further propose rank-preserving fine-tuning (RPFT), which adapts only a small subset of retained singular components so that the allocated rank stays bounded without re-decomposition. Experimental results show that UniRank cuts zero-shot perplexity by up to 50\%, improves average reasoning accuracy by 3.0\% over LoRAP at 25\% sparsity, and boosts four SVD-based decomposition methods as a plug-and-play module.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2606.21847 [cs.LG]

(or arXiv:2606.21847v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.21847 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.