Pull, Don't Recompute: Peer-to-Peer KV Cache Sharing in llm-d

A request arrives and its 48K-token prefix is already sitting in a KV cache, but the endpoint holding it - one schedulable serving instance: either a complete engine instance or a rank within a multi-rank deployment - is busy. Route the request to the cache owner and it queues behind that backlog; route it to an idle endpoint and it spends seconds recomputing work the cluster already did. Waiting and recomputing are both wrong answers. P2P (peer-to-peer) KV cache sharing adds the third option: send the request to the best endpoint for load, and copy the finished KV to it. A busy owner is only one way locality breaks. Load balancing may select another endpoint, or the required KV may have been generated by a different serving role. In every such case the llm-d router already knows how much of a request's prefix each candidate endpoint holds. With P2P, that knowledge becomes a transfer instruction: route the request to the best endpoint overall, then tell it where to fetch the missing prefix. On GLM-5.2-FP8 at concurrency 64, P2P increased successful throughput (requests completing end-to-end within the fixed 300-second window) with both prefix-cache routing methods: 4.6% with approximate routing and 9.7%…