analiza aiqbench
This research introduces GA-SVD, a novel compression framework that optimizes low-rank decomposition specifically for preserving generative capabilities in large language models. By leveraging on-policy knowledge distillation and evolutionary rank allocation, the method achieves state-of-the-art results across diverse model architectures without relying on heuristic assumptions. The approach effectively addresses the trade-off between model size reduction and performance retention on generation tasks, offering a practical path for deploying LLMs in resource-constrained environments.
GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression
arXiv:2610.15398v1 Announce Type: new Abstract: Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.
Abstract:Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.
Machine Learning (cs.LG)
arXiv:2610.15398 [cs.LG]
(or arXiv:2610.15398v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.15398 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte
Subiecte conexe
Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering
arXiv · 15 iun. 2026
Simulating Quantum Walk Hamiltonians without Pauli Decomposition
arXiv · 9 iun. 2026
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
arXiv · 8 iun. 2026
Full-Model Optimality for Tunable Linear Generative Priors in Compressed Sensing
arXiv · 3 sept. 2026