analiza aiqbench
VECTOR addresses the KV cache bottleneck in long-context LLM inference by introducing a three-way token routing strategy that recovers reconstructable information lost under binary eviction. This plug-and-play augmentation significantly outperforms existing compression methods on benchmarks like LongBench and RULER while incurring no overhead during the decoding phase. The method effectively balances exact retention with approximation to optimize memory usage without sacrificing model stability.
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
arXiv:2605.23258v2 Announce Type: replace Abstract: KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. On LongBench, RULER with contexts up to 128K tokens, and multi-turn SCBench, VECTOR yields significant gains over its base eviction methods under high compression. At a 90% compression ratio, SnapKV augmented with VECTOR outperforms six competing compression methods, including D2O and RocketKV, on all three evaluated models. Since VECTOR makes all routing decisions at prefill, decoding runs on a standard compressed cache with no measurable overhead.
Abstract:KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. On LongBench, RULER with contexts up to 128K tokens, and multi-turn SCBench, VECTOR yields significant gains over its base eviction methods under high compression. At a 90% compression ratio, SnapKV augmented with VECTOR outperforms six competing compression methods, including D2O and RocketKV, on all three evaluated models. Since VECTOR makes all routing decisions at prefill, decoding runs on a standard compressed cache with no measurable overhead.
Machine Learning (cs.LG)
arXiv:2605.23258 [cs.LG]
(or arXiv:2605.23258v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.23258 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 260 cuvinte
Subiecte conexe
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
arXiv · 6 oct. 2026
Beyond RAG: Task-aware knowledge compression for enterprise AI on AWS
AWS · 27 iul. 2026
ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics
arXiv · 9 sept. 2026
Serving DeepSeek-V4: why million-token context is an inference systems problem
Together AI · 11 mai 2026