aiqbench analysis
This research investigates the trade-offs between cost-efficiency and factual accuracy when language models utilize precomputed memory via saved key-value caches. The findings indicate that while this approach reduces context re-feeding, it introduces significant degradation risks when memories are assembled from disparate parts or fail to incorporate real-time corrections. For deployed systems, the study suggests a hybrid strategy of periodic full rebuilds supplemented by targeted cache injections to maintain information currency without prohibitive computational overhead.
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
arXiv:2608.30647v1 Announce Type: new Abstract: Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
Abstract:Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2608.30647 [cs.CL]
(or arXiv:2608.30647v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.30647 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 310 words
Related coverage
Intelligence is Free, Now What? <br> Data Systems for, of, and by Agents
BAIR Blog · Jul 7, 2026
Does preservation make sense before we know how to revive?
LessWrong · Jun 15, 2026
From Vector Search to Agentic RAG: Building an Enterprise Research Analyst with hipVS
AMD · Jul 15, 2026
From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon
BAIR Blog · Jul 29, 2026