aiqbench analysis
This research provides a rigorous benchmark comparing three distinct methods for injecting new information into LLMs: context window documents, latent representations (KV-cache), and parametric fine-tuning. The study reveals that while KV-cache based approaches like Cartridges offer superior accuracy in retrieval scenarios, they introduce specific risks such as catastrophic forgetting, whereas parametric methods face significant efficiency penalties at high compression rates. For practitioners optimizing knowledge-intensive applications, these findings highlight the critical trade-offs between storage budgets, performance gains, and model stability when selecting an adaptation strategy.
Where Should a Document Live: Context, Representations, or Parameters?
arXiv:2609.17346v1 Announce Type: cross Abstract: To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.
Title:Where Should a Document Live: Context, Representations, or Parameters?
Abstract:To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.17346 [cs.CL]
(or arXiv:2609.17346v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.17346 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 287 words
Related coverage
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment
Latent Space · Oct 16, 2026
Best Embedding Models in 2026
OpenRouter · Sep 23, 2026
Performance Profiling on AMD GPUs – Part 5: Profiling-Driven Kernel Optimization with an AI Code-Assist Tool
AMD · Jul 16, 2026
Benchmarking Kimi-K3 Across vLLM, SGLang, and ATOM on MI350X
AMD · Sep 22, 2026