analiza aiqbench
CateKV addresses the memory bottleneck in long-context LLM inference by exploiting sequential consistency patterns within attention heads to optimize KV cache management. This approach achieves significant reductions in memory usage and latency without compromising accuracy, offering a practical pathway for scaling context windows in production environments.
CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
arXiv:2608.30295v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to $2.72\times$ and accelerates decoding by $2.18\times$ in single-sample inputs, and boosts throughput by $3.96\times$ in batch scenarios.
Abstract:Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to $2.72\times$ and accelerates decoding by $2.18\times$ in single-sample inputs, and boosts throughput by $3.96\times$ in batch scenarios.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2608.30295 [cs.LG]
(or arXiv:2608.30295v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.30295 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 246 cuvinte
Subiecte conexe
Parallelize speculative decoding with P-EAGLE on Amazon SageMaker AI
AWS · 16 iun. 2026
Netskope (NTSK) Q2 2027 Earnings Call Transcript
fool.com · 10 sept. 2026
TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
arXiv · 1 sept. 2026
Parallelization constraints could delay a technological singularity [Linkpost]
LessWrong · 31 iul. 2026