aiqbench analysis
This research proposes a novel layer-wise design space for KV cache eviction, challenging the standard practice of applying uniform retention rules across all Transformer layers. By systematically composing existing methods and optimizing their placement based on offline profiles, the study demonstrates that heterogeneous routing significantly outperforms homogeneous policies and standalone baselines in long-context benchmarks. This work establishes layer-specific compression strategies as a critical optimization dimension for improving LLM inference efficiency without increasing cache budgets.
Exploring a Layer-Wise Design Space for KV Cache Eviction
arXiv:2606.15157v2 Announce Type: replace-cross Abstract: KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Abstract:KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.15157 [cs.LG]
(or arXiv:2606.15157v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.15157 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 272 words
Related coverage
Presentation: Adaptive Recommenders in the Real World: Inference, Evals, and System Design
InfoQ · Sep 26, 2026
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
arXiv · Jun 16, 2026
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · Jun 11, 2026
A kernel-centric path to real-time video generation on Trainium
Amazon Science · Sep 25, 2026