aiqbench analysis
This paper tackles a practical inference problem: reusing cached attention states for retrieved documents in RAG when those documents appear in a different order or context than when they were first computed. Its main finding is that naive composition preserves only about 38-47% of full-prefill answers on two QA benchmarks, while re-encoding a causal suffix from a selected start document recovers most of that behavior. The efficiency gain is limited, though: the authors report that faithful repair re-encodes most of the context, so their single-request L40S study favors full prefill across workloads. Readers building RAG serving stacks should treat the speedup claims as conditional on the repair cost, not as a general win.
SAFE-KV: Causal Suffix Repair for Behavior-Preserving Non-Prefix KV Cache Reuse in RAG
arXiv:2610.15186v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) reuses the same documents across many requests, making non-prefix KV-cache reuse attractive. Yet a cached document state is not reusable text: it is a context-conditioned causal state whose keys and values depend on the prefix, position, and preceding documents under which it was computed. We formulate non-prefix KV reuse as a behavior-preservation problem with full prefill on the exact prompt as the reference, and introduce SAFE-KV, which repairs a composed cache by re-encoding the causal suffix from a selected start document. On Qwen2.5-7B-Instruct, prefix-conditioned composition preserves only 47.2% of HotpotQA-500 and 37.8% of 2Wiki-500 full-prefill answers; SAFE-KV with risk-guided K2 selection restores 99.8%/98.4% (donor-copy protocol) and 100.0%/98.4% with actual suffix forwarding. Document-only and fixed-window repairs recover far less, because the document states after the repaired region keep histories that omit it. Reuse without repair is exact up to four documents and degrades at sixteen, where suffix repair recovers 98-100%. Faithful repair re-encodes most of the context, so in a single-request L40S study SAFE-KV's operating rule serves every workload by full prefill. SAFE-KV thus establishes the causal-state contract for non-prefix reuse and a validated repair mechanism that meets it.
Abstract:Retrieval-augmented generation (RAG) reuses the same documents across many requests, making non-prefix KV-cache reuse attractive. Yet a cached document state is not reusable text: it is a context-conditioned causal state whose keys and values depend on the prefix, position, and preceding documents under which it was computed. We formulate non-prefix KV reuse as a behavior-preservation problem with full prefill on the exact prompt as the reference, and introduce SAFE-KV, which repairs a composed cache by re-encoding the causal suffix from a selected start document. On Qwen2.5-7B-Instruct, prefix-conditioned composition preserves only 47.2% of HotpotQA-500 and 37.8% of 2Wiki-500 full-prefill answers; SAFE-KV with risk-guided K2 selection restores 99.8%/98.4% (donor-copy protocol) and 100.0%/98.4% with actual suffix forwarding. Document-only and fixed-window repairs recover far less, because the document states after the repaired region keep histories that omit it. Reuse without repair is exact up to four documents and degrades at sixteen, where suffix repair recovers 98-100%. Faithful repair re-encodes most of the context, so in a single-request L40S study SAFE-KV's operating rule serves every workload by full prefill. SAFE-KV thus establishes the causal-state contract for non-prefix reuse and a validated repair mechanism that meets it.
Accepted to Findings of AACL-IJCNLP 2026. 36 pages, 5 figures, 33 tables. Code: this https URL
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.15186 [cs.CL]
(or arXiv:2610.15186v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.15186 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 305 words
Related coverage
Why do models task game?
AI Alignment Forum · Aug 6, 2026
SCIT: Testing Causal Cache Carriers in Latent Chain-of-Thought Models
arXiv · Aug 28, 2026
Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation
arXiv · Jul 31, 2026
PatchKV: Efficient KV Cache Recovery for Dynamically Edited LLM Contexts
arXiv · Sep 23, 2026