NVIDIA boosts MoE AI training 10.4x with JAX and Transformer Engine

NVIDIA has achieved a 10.4x improvement in the performance of Mixture of Experts (MoE) AI training, jumping from 103 TFLOPS/GPU to 1,068 TFLOPS/GPU on NVIDIA GB200 GPUs, the company says. The leap resulted from combining the JAX Python library with NVIDIA Transformer Engine, tackling a significant bottleneck in scaling this increasingly popular AI architecture. Researchers found that, in an unoptimized DeepSeek-V3 MoE training run, inter-GPU communication consumed 84% of accumulated kernel time. Solving the challenge required specialized kernels to handle ragged tensors created when a learned router distributes tokens unevenly across expert networks. NVIDIA and its partners achieved a 10.4x improvement in TFLOPS/GPU. This substantial gain stems from targeted kernel optimizations within Transformer Engine, specifically designed to address bottlenecks unique to MoE…