analiza aiqbench
VaSE introduces a training-free eviction strategy that protects high-magnitude value states to prevent catastrophic reasoning loops while leveraging stochasticity for better cache diversity. This approach achieves superior accuracy over current selection-based methods with 4x compression on Qwen3 models, effectively bridging the efficiency-accuracy gap for long-context reasoning tasks. The technique is compatible with FlashAttention2 and enables a static memory footprint, offering a practical path forward for deploying memory-constrained reasoning models.
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
arXiv:2606.03928v1 Announce Type: new Abstract: Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
Abstract:Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2606.03928 [cs.LG]
(or arXiv:2606.03928v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.03928 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix
arXiv · 29 sept. 2026
AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning
arXiv · 6 oct. 2026
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · 11 iun. 2026
Why do models task game?
AI Alignment Forum · 6 aug. 2026