analiza aiqbench
This research introduces Hybrid Latent Attention (HLA) to address the memory bottleneck in looped language models, which repeat layers to deepen context without increasing parameters. By storing older tokens as compact latents rather than full key-value pairs, HLA reduces cache size by over 10x while maintaining high accuracy on math and reasoning benchmarks. The method enables significantly higher decoding throughput and concurrent sequence capacity on GPUs, offering a practical path forward for scaling long-context AI inference.
Hybrid Latent Attention for Looped Language Models
arXiv:2610.07940v1 Announce Type: new Abstract: Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.
Abstract:Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.07940 [cs.CL]
(or arXiv:2610.07940v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.07940 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
arXiv · 12 oct. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026
[AINews] not much happened today
Latent Space · 18 sept. 2026