Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni

Out-of-the-box version: For the DLO + AllGather quickstart below, use vLLM 0.27.0 with vLLM-Omni v0.27.0rc1. vLLM-Omni's Distributed Layerwise Offload enables video generation models larger than single-device HBM (e.g., Cosmos3-Super 64B / 124 GB) to run across multiple NPUs or GPUs with minimal host memory overhead. The stack includes: Meta-device initialization + mmap weight loading: Weights are loaded as mmap views pointing to shared OS page cache, eliminating O(dp_size × model_size) RSS during model creation. Cold-start cgroup-visible peak drops by 73% (178 GB → 47 GB for Cosmos3-Nano DP4). Weight sharding + AllGather: Each rank stores only 1/dp_size of the model. Full layer weights are reconstructed at runtime via AllGather, overlapped with computation on dedicated streams. Fixed double-buffer scheme: Exactly 2 layers of weights reside on each device at any time, independent of the total layer count. Buffer capacity still scales with the model's largest block, and total HBM also includes workload-dependent activation and communication buffers. In the measured 720p 10s workload, peak HBM grew about 22% (23.1 → 28.1 GB) from the 17B to the 64B model; idle HBM grew about 27% (11.5 → 14.6 GB). DP multi-concurrency: Each DP rank processes a different request in parallel, achieving 3.3× throughput…