analiza aiqbench
UniSVQ addresses the trade-off between scalar and vector quantization by unifying them into a 2-bit framework that maintains compatibility with optimized integer kernels while reducing storage overhead. The method achieves superior inference throughput and performance comparable to advanced vector quantization techniques through a data-driven fine-tuning strategy that minimizes reconstruction error. This approach offers a practical pathway for deploying large language models at lower costs without sacrificing accuracy.
UniSVQ: 2-bit Unified Scalar-Vector Quantization
arXiv:2606.10520v1 Announce Type: new Abstract: Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.
Abstract:Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.
Computation and Language (cs.CL)
arXiv:2606.10520 [cs.CL]
(or arXiv:2606.10520v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10520 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 222 cuvinte
Subiecte conexe
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · 11 iun. 2026
A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization
arXiv · 3 sept. 2026
Performance Profiling on AMD GPUs – Part 5: Profiling-Driven Kernel Optimization with an AI Code-Assist Tool
AMD · 16 iul. 2026
RiboSphere: Learning Unified and Efficient Representations of RNA Structures
arXiv · 6 aug. 2026