analiza aiqbench
This paper proposes a training-free post-training quantization method that reshapes LLM activation distributions so they fit a low-bit uniform quantizer better, rather than only suppressing outliers or minimizing reconstruction error. The authors report that under W4A4KV4 it retains about 97% of floating-point accuracy on average and narrows the LLaMA-2 13B gap by 42% versus the prior state of the art. For readers deploying LLMs, the key questions are whether these gains hold on newer model families and real hardware kernels, and what the latency and memory tradeoffs look like outside the paper's benchmarks.
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
arXiv:2605.26175v2 Announce Type: replace Abstract: Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at [https://github.com/LLIKKE/InfoQuant](https://github.com/LLIKKE/InfoQuant)
Abstract:Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at [this https URL](this https URL)
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.26175 [cs.LG]
(or arXiv:2605.26175v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.26175 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 308 cuvinte
Subiecte conexe
Minimum Distortion Quantization with Specified Output Distribution
arXiv · 10 iun. 2026
Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration
arXiv · 7 oct. 2026
RDQ: Residual Distribution Quantization for Large Language Models
arXiv · 29 iul. 2026
SoloQ: Calibration-Free Quantization for Diffusion Language Models
arXiv · 7 oct. 2026