aiqbench analysis
MPK introduces a novel compiler and runtime that fuses multi-GPU inference into a single mega-kernel, enabling fine-grained overlap of computation and communication at the streaming multiprocessor level. By automating these low-level optimizations, the system achieves up to 1.7x lower latency compared to conventional kernel-per-operator approaches while maintaining developer flexibility. This work represents a significant step toward extracting near-hardware-limit performance for large language model serving without requiring manual kernel rewriting.
MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs
arXiv:2512.22219v2 Announce Type: replace-cross Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel. MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors (SMs), enabling cross-operator software pipelining, \rev{fine-grained overlap of computation and communication, and other optimizations that are infeasible under the conventional kernel-per-operator execution model}. The MPK compiler lowers tensor programs into optimized SM-level task graphs and generates fast CUDA implementations for each task, while the MPK in-kernel parallel runtime executes these tasks within a single persistent mega-kernel using decentralized scheduling across SMs. Together, these components provide end-to-end kernel fusion with minimal developer effort, while preserving the flexibility of existing programming models. Our evaluation shows that MPK significantly outperforms existing kernel-per-operator LLM serving systems, achieving up to 1.7$\times$ lower end-to-end inference latency and pushing LLM inference performance close to the limits of the underlying hardware. MPK is publicly available at https://github.com/mirage-project/mirage.
Abstract:We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel. MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors (SMs), enabling cross-operator software pipelining, \rev{fine-grained overlap of computation and communication, and other optimizations that are infeasible under the conventional kernel-per-operator execution model}. The MPK compiler lowers tensor programs into optimized SM-level task graphs and generates fast CUDA implementations for each task, while the MPK in-kernel parallel runtime executes these tasks within a single persistent mega-kernel using decentralized scheduling across SMs. Together, these components provide end-to-end kernel fusion with minimal developer effort, while preserving the flexibility of existing programming models. Our evaluation shows that MPK significantly outperforms existing kernel-per-operator LLM serving systems, achieving up to 1.7$\times$ lower end-to-end inference latency and pushing LLM inference performance close to the limits of the underlying hardware. MPK is publicly available at this https URL.
Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Programming Languages (cs.PL)
arXiv:2512.22219 [cs.DC]
(or arXiv:2512.22219v2 [cs.DC] for this version)
Full text under the publisher's open licence · 181 words
Related coverage
NVIDIA CUDA 13.3 Enhances GPU Development with Tile Programming in C++, Compiler Autotuning, and Python Updates
NVIDIA · May 26, 2026
Introducing CUDA Rust: Two Tracks for Writing GPU Kernels
NVIDIA · Sep 8, 2026
Inside the megakernel serving engine for North Mini Code
Cohere · Sep 8, 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · May 11, 2026