aiqbench analysis
This paper proposes TVQ, a method that makes vector-quantized LLM weights trainable with straight-through estimation, rather than limiting them to layer-wise reconstruction. The reported result is near-lossless 2-bit quantization of LLaMA2-70B with about a 1% accuracy drop on QA benchmarks, but these are authors' claims that need independent replication. For practitioners, the practical question is whether the training budget and PyTorch implementation are light enough to adopt over existing training-free methods.
Trainable Vector Quantization for Large Language Models
arXiv:2610.41822v1 Announce Type: new Abstract: Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.
Abstract:Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.
Machine Learning (cs.LG)
arXiv:2610.41822 [cs.LG]
(or arXiv:2610.41822v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.41822 arXiv-issued DOI via DataCite (pending registration)
Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2026. Lecture Notes in Computer Science, vol 16944, pp 245-259. Springer, Cham
https://doi.org/10.1007/978-3-032-37667-1_15 DOI(s) linking to related resources
Full text under the publisher's open licence · 218 words
Related coverage
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
arXiv · Jun 10, 2026
Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
arXiv · Sep 1, 2026
Predicate Neuron: Differentiable Quantization of Predicate Relations in Gradient-based Learning
arXiv · Oct 31, 2026
Revisiting Error Feedback for Model Quantization
arXiv · Oct 14, 2026