analiza aiqbench
This research introduces Concentration-Alignment Transforms (CAT) to address accuracy loss in post-training quantization for large language models. By jointly optimizing weight/activation concentration and alignment, the method achieves 4-bit precision that matches or exceeds existing transform-based techniques without requiring complex hardware changes. This advance offers a practical pathway to deploying efficient LLMs on resource-constrained devices while maintaining high fidelity.
Dissecting Quantization Error: A Concentration-Alignment Perspective
arXiv:2603.04359v2 Announce Type: replace-cross Abstract: Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.
Abstract:Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.04359 [cs.LG]
(or arXiv:2603.04359v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.04359 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · 30 iul. 2026
The basic argument for making AI better at moral philosophy
LessWrong · 14 oct. 2026
Our response to Séb Krier on Plan A
LessWrong · 14 iul. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026