aiqbench analysis
This research addresses the critical bottleneck of long-context inference by introducing Jacap, a KV cache eviction method grounded in local information geometry rather than empirical heuristics. By modeling attention as a nonlinear Gaussian channel and deriving Jacobian Information Capacity, the approach rigorously captures token utility under softmax dynamics to enable superior compression without sacrificing performance.
Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
arXiv:2609.08131v1 Announce Type: new Abstract: Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.
Abstract:Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.
Computation and Language (cs.CL)
arXiv:2609.08131 [cs.CL]
(or arXiv:2609.08131v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.08131 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 228 words
Related coverage
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · Jun 11, 2026
Physics-Informed Stochastic Configuration Machine: A Backpropagation-Free Neural Network with Fast Training for Nonlinear Differential Equations
arXiv · Aug 28, 2026
Trajectory-Aware Node Contributions and the Limits of Static Controllability
arXiv · Jun 3, 2026
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face · Sep 10, 2026