aiqbench takeChips & Infra

This article details the infrastructure challenges of serving Moonshot AI's 2.8T-parameter Kimi K3.1 Pro model, specifically addressing distributed orchestration for massive MoE architectures and long-context windows across TPUs and GPUs. It presents a Kubernetes-native inference control plane solution that enables state-aware routing and memory offloading to bypass hardware limitations inherent in trillion-parameter models.

Sources coveringllm-d, Google Cloud Blog, Import AI

Serving Moonshot AI's Kimi K3.1 Pro with llm-d across TPUs and GPUs

llm-dNovember 4, 20266 min read

Day-0 distributed serving recipes for Moonshot AI's 2.8T-parameter Kimi K3.1 Pro at 1M context using llm-d across Ironwood TPU v7x (SGLang) and NVIDIA GB300 GPUs (vLLM), on Google Cloud.

Running multi-trillion-parameter Mixture-of-Experts (MoE) models in production shifts the primary engineering bottleneck from model weights to distributed systems orchestration. With Moonshot AI's Kimi K3.1 Pro (moonshotai/Kimi-K3.1-Pro), infrastructure engineers must accommodate a ~2.8-trillion-parameter footprint (~104B active parameters per token across 896 routed experts), a 1M-token context window, and a hybrid 93-layer attention stack (69 Kimi Delta Attention layers paired with 24 Gated Multi-head Latent Attention layers). At this footprint, the model weights alone surpass the High-Bandwidth Memory (HBM) capacity of any single machine, requiring multi-node Tensor and Expert Parallelism across high-bandwidth interconnects. Compounding the hardware footprint, long-horizon agentic sessions expose the limits of stateless Kubernetes load balancing: 1M-token KV caches rapidly exhaust accelerator HBM,…

Excerpt shown under the publisher's RSS terms · 113 words · full text and figures are on the publisher's site

Read at llm-d ↗Ask about this article