aiqbench analysis
This paper targets the latency of sparse KV-cache selection in long-context LLM decoding. It moves the routing step off the critical path by using proxy queries from earlier hidden states, and reports small quality differences on LongBench, RULER, and NIAH/Passkey, plus up to 12.4% higher output throughput in a TensorRT-LLM setup with CPU-offloaded routing. The results are from specific methods, models, and settings, so readers should check whether they carry over to their own workloads before treating the throughput gains as general.
EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies
arXiv:2611.01715v1 Announce Type: new Abstract: Query-conditioned sparse KV selection reduces long-context inference cost by attending to a subset of historical KV entries at each decoding step. Existing selectors must wait for the target-layer query before scoring candidates and generating indices, placing routing on the immediate pre-attention path. We introduce EarlySelectKV, a training-free transformation that uses proxy queries from earlier hidden states to precompute selected KV indices while retaining the real target-layer query for final attention. Tests with RocketKV, Quest, and Loki show no consistent pattern of quality degradation across the evaluated settings. In the primary RocketKV evaluation, ES-RocketKV differs by +0.018 points on LongBench, approximately -0.01 points on RULER, and +0.25 percentage points on representative NIAH/Passkey settings. In a TensorRT-LLM deployment with CPU-offloaded routing, ES-RocketKV improves end-to-end output throughput by up to 12.4% and reduces GPU-resident memory by 11.1% of the combined KV/K-summary pool. This makes routing timing an explicit optimization target alongside KV subset selection in sparse long-context decoding. Code is available at https://github.com/sea-with-sakura/EarlySelectKV.
Read the full article at arXiv