Serving Moonshot AI's Kimi K3.1 Pro with llm-d across TPUs and GPUs

Running multi-trillion-parameter Mixture-of-Experts (MoE) models in production shifts the primary engineering bottleneck from model weights to distributed systems orchestration. With Moonshot AI's Kimi K3.1 Pro (moonshotai/Kimi-K3.1-Pro), infrastructure engineers must accommodate a ~2.8-trillion-parameter footprint (~104B active parameters per token across 896 routed experts), a 1M-token context window, and a hybrid 93-layer attention stack (69 Kimi Delta Attention layers paired with 24 Gated Multi-head Latent Attention layers). At this footprint, the model weights alone surpass the High-Bandwidth Memory (HBM) capacity of any single machine, requiring multi-node Tensor and Expert Parallelism across high-bandwidth interconnects. Compounding the hardware footprint, long-horizon agentic sessions expose the limits of stateless Kubernetes load balancing: 1M-token KV caches rapidly exhaust accelerator HBM,…