The Four Caches in LLM Serving

As LLM applications grow more complex, inference cost and latency become increasingly important. A single request can contain thousands or even millions of tokens from system instructions, conversation history, retrieved documents, tool definitions, and user input. Reprocessing the same information again and again wastes both time and compute. Caching helps avoid this repeated work. But LLM caching isn’t a single technique. Different caches operate at different stages of the serving stack and solve different problems. In this article, we’ll explore four key techniques: KV caching, prefix caching, prompt caching, and semantic caching. What Exactly Is Being Cached? 1. KV Cache: Remembering What the Model Has Already Processed Let’s start with the cache that is fundamental to every modern autoregressive LLM inference: the KV cache. When the LLM generates a response, it doesn’t produce the entire response in one shot. It generates one token at a time autoregressively. For example, if the model is generating the sentence “Quantum computing is a new approach to computation,” the model might generate it approximately as: “Quantum” → “computing” → “is” → “a” → “new” → “approach” → … and so on. At every generation step, the Transformer uses its attention mechanism to determine how…