NVIDIA’s EPD for multimodal AI

Vision encoding within multimodal AI can require hundreds of milliseconds, creating a bottleneck that impacts overall response times. To address this, NVIDIA researchers Jesse Gu, Ryan McCormick and Akshatha Kamath have detailed how separating the vision encoder stage from the prefill and decode phases, a technique called Encode-Prefill-Decode (EPD) disaggregation, can deliver significant performance gains, the company says. Utilizing EPD disaggregation with NVIDIA Dynamo achieves up to 5x faster time to first token and 7x faster end-to-end response time, according to the company. This optimization is particularly effective for image-heavy prompts and quantized mixture-of-experts models, allowing for independently scalable stages. Efficiently serving multimodal AI requests demands careful allocation of computational resources, and a new technique called Encode-Prefill-Decode (EPD) disaggregation offers substantial gains in speed when implemented with NVIDIA Dynamo. By routing both request types to a prefill worker but invoking a separate encoder pool only for multimodal requests needing embeddings, EPD allows text-only prompts to proceed immediately. Jesse Gu, a summer systems software engineering intern on the Dynamo team, explains that this separation is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts (MoE) models. Hardware configuration plays an important role in maximizing the benefits of EPD disaggregation; the…