aiqbench analysis
SigmaScale introduces a novel approach to LLM compression by learning activation-aware scaling matrices that optimize truncated SVD decomposition, effectively lowering the intrinsic rank of weight matrices. Experimental results on Llama 3.1 and Qwen3-8B demonstrate that this method achieves performance competitive with existing state-of-the-art techniques while offering a more flexible adaptation to individual model structures. This technique presents a viable pathway for reducing inference computing costs without sacrificing model capability on key benchmarks.
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
arXiv:2606.07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
Abstract:We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2606.07098 [cs.CL]
(or arXiv:2606.07098v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.07098 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 246 words
Related coverage
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · Aug 18, 2026
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
arXiv · Aug 3, 2026
Dynamics as Code: On Model Compression via Dynamic System
arXiv · Oct 9, 2026