aiqbench analysis
COEC introduces a training-free compensation framework for structured pruning in large language models, utilizing alternating orthogonal rotations and singular value rescaling to mitigate accuracy loss without retraining. By optimizing on a reduced Stiefel manifold and tempering calibration matrices, this method effectively recovers performance across various sparsity levels in models like Llama-3 and Qwen2.5. This advancement offers a practical pathway for deploying more efficient LLMs while maintaining high fidelity through advanced geometric adjustments to weight columns.
COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models
arXiv:2608.21142v1 Announce Type: new Abstract: Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention projections.All components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.
Abstract:Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention this http URL components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.
Machine Learning (cs.LG)
arXiv:2608.21142 [cs.LG]
(or arXiv:2608.21142v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.21142 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 332 words
Related coverage
Quantization and Pruning Methods to Make Your LLM Leaner
KDnuggets · Aug 28, 2026
SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning
arXiv · Oct 15, 2026
The Residual Stream's Effective Depth
arXiv · Sep 28, 2026
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Hugging Face · Sep 21, 2026