analiza aiqbench
PhaseQuant addresses a critical bottleneck in 4-bit LLM inference by leveraging residual-stream dynamics to identify layers prone to quantization collapse, rather than relying solely on static smoothing techniques. By dynamically assigning higher precision only where necessary, the method preserves model accuracy and throughput under aggressive mixed-precision constraints without requiring retraining. This approach offers a practical pathway for deploying efficient large language models in resource-constrained environments.
PhaseQuant: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics
arXiv:2606.12487v2 Announce Type: replace Abstract: Post-training quantization (PTQ) is essential for efficient large language model (LLM) inference, but reliably quantizing activations remains challenging when weights, activations, and KV caches are all quantized to 4-bit precision. A key difficulty lies in massive activations, whose extreme values dominate the activation range and amplify quantization error. State-of-the-art methods mitigate massive activations through transformation-based smoothing, such as orthogonal rotations and affine scaling, but overlook the cross-layer dynamics of the residual stream. In this paper, we show that massive activations emerge and disappear in a phase-wise pattern across network depth, injecting abrupt updates into the residual stream. These updates cause the newly injected layer-wise signal to dominate the 4-bit quantization scale and weaken the accumulated residual information. To characterize this behavior, we introduce two metrics, the Update Dominance Ratio (UDR) and the Historical Retention Score (HRS), and show that static transformation-based smoothing cannot fully resolve the resulting dynamic quantization instability. Based on this analysis, we propose PhaseQuant, a phase-aware mixed-precision post-training quantization policy for activation quantization. PhaseQuant identifies quantization-sensitive layers from residual-stream dynamics and assigns 8-bit activation precision only to these layers, while keeping weights, KV caches, and all remaining activations at 4-bit precision. It can be directly integrated with strong PTQ baselines such as QuaRot, SpinQuant, and FlatQuant. Experiments on LLaMA-2 and LLaMA-3 show that PhaseQuant consistently improves perplexity and zero-shot QA performance under W4A4KV4 quantization, while achieving 1.05$\times$--1.07$\times$ throughput improvement with modest memory overhead. These results demonstrate a practical path toward robust low-bit LLM inference.
Abstract:Post-training quantization (PTQ) is essential for efficient large language model (LLM) inference, but reliably quantizing activations remains challenging when weights, activations, and KV caches are all quantized to 4-bit precision. A key difficulty lies in massive activations, whose extreme values dominate the activation range and amplify quantization error. State-of-the-art methods mitigate massive activations through transformation-based smoothing, such as orthogonal rotations and affine scaling, but overlook the cross-layer dynamics of the residual stream. In this paper, we show that massive activations emerge and disappear in a phase-wise pattern across network depth, injecting abrupt updates into the residual stream. These updates cause the newly injected layer-wise signal to dominate the 4-bit quantization scale and weaken the accumulated residual information. To characterize this behavior, we introduce two metrics, the Update Dominance Ratio (UDR) and the Historical Retention Score (HRS), and show that static transformation-based smoothing cannot fully resolve the resulting dynamic quantization instability. Based on this analysis, we propose PhaseQuant, a phase-aware mixed-precision post-training quantization policy for activation quantization. PhaseQuant identifies quantization-sensitive layers from residual-stream dynamics and assigns 8-bit activation precision only to these layers, while keeping weights, KV caches, and all remaining activations at 4-bit precision. It can be directly integrated with strong PTQ baselines such as QuaRot, SpinQuant, and FlatQuant. Experiments on LLaMA-2 and LLaMA-3 show that PhaseQuant consistently improves perplexity and zero-shot QA performance under W4A4KV4 quantization, while achieving 1.05$\times$--1.07$\times$ throughput improvement with modest memory overhead. These results demonstrate a practical path toward robust low-bit LLM inference.
Machine Learning (cs.LG)
arXiv:2606.12487 [cs.LG]
(or arXiv:2606.12487v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.12487 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 339 cuvinte
Subiecte conexe
Optimizing ATOM and vLLM-ATOM for High-Interactivity Inference
AMD · 1 sept. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Dropless MoE Training in JAX with Primus-Turbo
AMD · 10 iun. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026