AI Inference & Efficiency Engineer

Two very different phases hide behind one API call

When you send a prompt to a language model and watch the answer stream back, the server is doing two jobs that have almost nothing in common. First it runs prefill: every token of your prompt goes through the network at once, as one big matrix multiplication per layer. Then it runs decode: one new token is produced per forward pass, fed back in, and the loop repeats until the model stops. Everything about inference cost follows from the fact that these two phases are bottlenecked by different parts of the hardware.

Prefill is compute-bound

In prefill the model multiplies a matrix of N prompt tokens by each weight matrix. The weights are read from memory once and reused N times, so the accelerator spends most of its time doing arithmetic. Prefill therefore scales with the number of prompt tokens and with the chip's peak FLOPs. Long prompts cost real compute, which is why providers charge for input tokens at all.

Decode is memory-bandwidth-bound

In decode the "matrix" of new tokens has one row. To produce one token the hardware must still stream every weight of the model from memory into the compute units, but it does only one multiply-add per weight. A modern GPU can perform hundreds of arithmetic operations in the time it takes to fetch one number from high-bandwidth memory, so during decode the arithmetic units mostly sit idle, waiting on memory. Horace He's "Making Deep Learning Go Brrrr" is the clearest short explanation of why: a program is either compute-bound, memory-bound or overhead-bound, and you cannot fix one with a tool meant for another.

The practical consequence is a rule of thumb you will use for the rest of this track: single-request decode speed is roughly memory bandwidth divided by model size in bytes. Halve the bytes and decode goes almost twice as fast; double the FLOPs of the chip and decode barely changes.

The KV cache: why long conversations get expensive

Attention needs, for every new token, the keys and values of every previous token in every layer. Recomputing them each step would make decode quadratic, so servers store them in a KV cache. Per token, the cache holds 2 (K and V) × layers × KV heads × head dimension × bytes per value. That memory grows linearly with context length and is multiplied by the number of concurrent requests, and it has to be read from memory on every decode step alongside the weights. Beyond a few thousand tokens of context the KV cache, not the weights, can dominate both memory capacity and decode bandwidth. The paper "Efficiently Scaling Transformer Inference" by Pope et al. works through exactly these costs for very large models and shows that KV-cache traffic is what forces techniques such as multi-query attention.

Worked example

Take a small decoder with 1 billion parameters, 16 layers, 8 KV heads of dimension 128, stored in 16-bit.

1. Weights: 1 × 10⁹ parameters × 2 bytes = 2 GB must be read per decode step.
2. KV cache per token: 2 × 16 layers × 8 heads × 128 × 2 bytes = 64 KB. An 8,000-token context costs 512 MB per request; sixteen such requests consume 8 GB — more than the weights.
3. On a laptop with 100 GB/s of memory bandwidth, the ceiling for one request is 100 GB/s ÷ 2 GB ≈ 50 tokens per second, before counting KV-cache reads. In practice you will see less; you will never see more.

Quantize the weights to 4 bits (0.5 GB) and the ceiling rises toward 200 tokens per second; the KV cache is unchanged unless you quantize it too. Run 16 requests at once and the weights are still read only once per step, so throughput scales almost 16× while per-request latency barely moves — the central insight of module 3. The inference chapter of "How to Scale Your Model" takes this arithmetic much further, including how attention FLOPs and sharding change the picture for large models.

What the model actually spends per token

A useful way to think about cost is bytes moved per generated token. Weights are a fixed cost per step; KV cache is a cost proportional to context; activations are small in decode. When someone proposes a speed-up, ask which of these it reduces. Quantization reduces weight bytes. Grouped-query attention and KV-cache quantization reduce cache bytes. Batching amortises weight bytes across requests. Speculative decoding produces several tokens per weight read. A faster matrix-multiplication kernel helps prefill and large-batch decode but does nothing for one user waiting on a long answer.

What is still unsettled

The picture above is a steady-state model; real serving mixes prefill and decode on the same chips and the interference between them is an active engineering topic (disaggregated prefill is one response). Compute-in-memory hardware and very low-bit formats could change where the bottleneck lies. Treat the rule of thumb as a first estimate to be checked by measurement, which is module 6's whole subject.

What to remember

• Prefill is compute-bound and scales with prompt length; decode is memory-bandwidth-bound and scales with model bytes.
• Single-request decode tokens/s ≈ memory bandwidth ÷ model size in bytes — an upper bound, not a promise.
• KV cache per token = 2 × layers × KV heads × head dimension × bytes; it grows with context and with concurrency, and it competes with weights for bandwidth.
• Every optimisation reduces one of: weight bytes, KV bytes, or how many tokens each weight read produces. Name which before you believe the claim.