aiqbench analysis
This is a serving-infrastructure update for vLLM, an open-source LLM inference engine. It shows that elastic expert-parallel scaling can cut downtime from roughly 100 seconds to under 2 seconds in one DeepSeek V4 Pro benchmark on GB200 GPUs. Readers running large MoE models should note that these figures come from a single vendor-run benchmark, so results on other models, hardware, or workloads may differ.
Achieving near-zero-downtime scaling in Elastic Expert Parallelism
How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM.
Traditionally, changing the number of GPUs in a vLLM service required adding or removing entire model replicas or restarting the engine with a different GPU count. As models grow larger, each replica needs more GPUs, so scaling by replica adds or removes GPUs in increasingly large increments. Restarting vLLM to scale DeepSeek V4 Pro from 16 to 20 GB200 GPUs stopped serving for 197 seconds (see the performance section for details), leaving those expensive GPUs allocated without generating tokens. In our first Elastic EP post, we showed how vLLM could add or remove GPUs from a running DEP deployment (data-parallel attention with expert-parallel MoE) without restarting it. While we avoided a restart, serving still stopped throughout reconfiguration as we initialized new workers, prepared the expanded communication groups, loaded model weights, and repeated kernel warmup and CUDA graph capture. In our DeepSeek V4 Pro benchmark, this pause lasted 100 seconds when scaling from 16 to 20 GPUs and 56 seconds when scaling back to 16 GPUs (full details in the performance section). In this post, we describe how asynchronous preparation and NIXL EP enabled Elastic EP reconfiguration with…
Excerpt shown under the publisher's RSS terms · 187 words · full text and figures are on the publisher's site