aiqbench analysis
mKernel addresses the critical communication bottleneck in distributed AI training by fusing computation with intra-node NVLink and inter-node RDMA operations at tile granularity. By adaptively partitioning GPU streaming multiprocessors between compute and communication roles, this library minimizes data movement across the network backend while supporting various parallelism strategies like tensor and expert parallelism. This approach demonstrates significant speedups on large-scale clusters without relying heavily on advanced hardware features like GPUDirect Async, offering a software-centric path to scaling LLM inference and training.
mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
arXiv:2609.13585v1 Announce Type: cross Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon as it is produced, but existing fused kernels are largely confined to a single NVLink domain. We present mKernel, a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity. mKernel partitions the streaming multiprocessors (SMs) of a persistent kernel into compute and communication roles, and an on-GPU controller tunes the SM partition adaptively at run time, since the best SM partition varies with the kernel and the input shape. It structures data movement hierarchically so that data traversing the inter-node network is minimized. Finally, it drives the network from the GPU through a lightweight command queue and host proxy implemented directly on RDMA verbs, which allows the same kernels to run on any network backend (e.g. InfiniBand and on AWS EFA); we observe, surprisingly, that GPUDirect Async (IBGDA) yields little additional benefit over host-assisted GPU-initiated communication. We implement five kernels spanning tensor, sequence, and expert parallelism. On two 16-GPU H200 clusters, mKernel achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88\times$ on Ring Attention.
Abstract:Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon as it is produced, but existing fused kernels are largely confined to a single NVLink domain. We present mKernel, a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity. mKernel partitions the streaming multiprocessors (SMs) of a persistent kernel into compute and communication roles, and an on-GPU controller tunes the SM partition adaptively at run time, since the best SM partition varies with the kernel and the input shape. It structures data movement hierarchically so that data traversing the inter-node network is minimized. Finally, it drives the network from the GPU through a lightweight command queue and host proxy implemented directly on RDMA verbs, which allows the same kernels to run on any network backend (e.g. InfiniBand and on AWS EFA); we observe, surprisingly, that GPUDirect Async (IBGDA) yields little additional benefit over host-assisted GPU-initiated communication. We implement five kernels spanning tensor, sequence, and expert parallelism. On two 16-GPU H200 clusters, mKernel achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88\times$ on Ring Attention.
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
arXiv:2609.13585 [cs.DC]
(or arXiv:2609.13585v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2609.13585 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 251 words
Related coverage
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · Oct 8, 2026
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · Sep 3, 2026
DI Series: Scaling GLM-5.1-FP8 to 64 MI300X GPUs
AMD · Aug 21, 2026
Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel
Hugging Face · Jun 24, 2026