analiza aiqbench
This paper shows that offline GPU kernel autotuning, the standard approach in LLM serving, misses a key dependency in Mixture-of-Experts layers: the best kernel configuration depends on how the router splits tokens across experts and GPUs, not just on batch size. RaMP chooses the configuration at runtime with a small on-GPU cost model. The reported gains (up to 1.45x on kernel speed and 1.11x on time per output token in vLLM) are measured against vLLM's own autotuner and backends, so they reflect serving-stack improvements rather than model quality. For inference engineers, the practical point is that MoE performance tuning may need to account for runtime routing distributions.
RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
arXiv:2604.26039v2 Announce Type: replace-cross Abstract: LLM serving systems autotune their GPU kernels offline by timing candidate kernel configurations, like tile sizes, at a range of batch sizes. This works when the number of tokens determines a kernel's work, as in dense layers. However, we show that it does not hold for Mixture-of-Experts (MoE) layers where the router decides how the tokens are split across experts and GPUs. We find that the best kernel configuration depends on this split, and not only on the number of tokens--under expert parallelism, choosing the configuration for each GPU's actual token distribution is up to 1.45x faster than the existing choice. The split is not known until the router runs, so it cannot be looked up in advance. Hence, we introduce RaMP, which chooses the kernel configuration at run time from the routing split itself. RaMP compiles top configurations into one polymorphic kernel, and a small cost model on the GPU picks among them in a few microseconds. On serving traffic its cost model was never fit on, the configurations RaMP's cost model chooses from the kernel's full configuration space are up to 1.45x faster than those vLLM's offline autotuner picks for the same kernel on batches of 128 tokens or more (1.27x over all batches), and within 3.5% of an exhaustive search that times every configuration. RaMP's kernel and choice together make the MoE layer up to 1.84x faster than vLLM's Triton MoE on batches of 128 tokens or more. In vLLM, under both tensor and expert parallelism, RaMP lowers time per output token by up to 1.11x over the fastest stock MoE backend.
Abstract:LLM serving systems autotune their GPU kernels offline by timing candidate kernel configurations, like tile sizes, at a range of batch sizes. This works when the number of tokens determines a kernel's work, as in dense layers. However, we show that it does not hold for Mixture-of-Experts (MoE) layers where the router decides how the tokens are split across experts and GPUs. We find that the best kernel configuration depends on this split, and not only on the number of tokens--under expert parallelism, choosing the configuration for each GPU's actual token distribution is up to 1.45x faster than the existing choice. The split is not known until the router runs, so it cannot be looked up in advance. Hence, we introduce RaMP, which chooses the kernel configuration at run time from the routing split itself. RaMP compiles top configurations into one polymorphic kernel, and a small cost model on the GPU picks among them in a few microseconds. On serving traffic its cost model was never fit on, the configurations RaMP's cost model chooses from the kernel's full configuration space are up to 1.45x faster than those vLLM's offline autotuner picks for the same kernel on batches of 128 tokens or more (1.27x over all batches), and within 3.5% of an exhaustive search that times every configuration. RaMP's kernel and choice together make the MoE layer up to 1.84x faster than vLLM's Triton MoE on batches of 128 tokens or more. In vLLM, under both tensor and expert parallelism, RaMP lowers time per output token by up to 1.11x over the fastest stock MoE backend.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
arXiv:2604.26039 [cs.LG]
(or arXiv:2604.26039v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.26039 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 366 cuvinte