aiqbench analysis
This research introduces a novel mixed-precision quantization method for hybrid language models that leverages observability Gramians to allocate bits dynamically without requiring calibration data or retraining. By combining variable bit widths with logarithmic decay rate quantization, the approach significantly reduces negative log-likelihood errors while maintaining low metadata overhead. The findings demonstrate substantial efficiency gains over existing baselines, offering a viable path toward deploying larger hybrid architectures on resource-constrained devices.
Low-Bit Recurrent States in Hybrid Language Models
arXiv:2609.30950v1 Announce Type: new Abstract: Hybrid language models maintain fixed-size recurrent states, but existing quantizers typically use eight bits or more. Quantization errors persist according to channel decay rates. We derive distortion weights from the observability Gramian and combine them with normalized state ranges for mixed-precision bit allocation, without calibration data, rotation, or training. We also quantize decay rates logarithmically. With per-token state quantization, a four-bit mean payload reduces excess negative log-likelihood by factors of 3.3--27.9 relative to the best of seven baselines across three hybrid models; metadata costs vary. At six bits, negative log-likelihood differs from the FP32-state baseline by less than 0.005 nats. Ablations separate gains from variable bit widths, decay weighting, and range normalization. With less frequent write-backs, gains diminish and depend on the model and budget.
Abstract:Hybrid language models maintain fixed-size recurrent states, but existing quantizers typically use eight bits or more. Quantization errors persist according to channel decay rates. We derive distortion weights from the observability Gramian and combine them with normalized state ranges for mixed-precision bit allocation, without calibration data, rotation, or training. We also quantize decay rates logarithmically. With per-token state quantization, a four-bit mean payload reduces excess negative log-likelihood by factors of 3.3--27.9 relative to the best of seven baselines across three hybrid models; metadata costs vary. At six bits, negative log-likelihood differs from the FP32-state baseline by less than 0.005 nats. Ablations separate gains from variable bit widths, decay weighting, and range normalization. With less frequent write-backs, gains diminish and depend on the model and budget.
9 pages, 2 figures, 3 tables. Submitted to ICASSP 2027
Machine Learning (cs.LG)
arXiv:2609.30950 [cs.LG]
(or arXiv:2609.30950v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.30950 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 229 words
Related coverage
M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
arXiv · Oct 19, 2026
LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
arXiv · Sep 24, 2026
Zyphra Release Zamba2-VL: Hybrid Mamba2–Transformer Vision-Language Models That Cut Time-to-First-Token by About an Order of Magnitude
MarkTechPost · Jun 12, 2026
CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models
arXiv · Sep 3, 2026