NVIDIA boosts BioNeMo models with efficient MoE training

NVIDIA is tackling a core challenge in scaling large language models: efficiency. Rather than processing every piece of data through every layer, mixture-of-experts (MoE) architectures activate only a select few subnetworks for each task. This approach promises greater capacity, but fragmented computation has historically limited gains; NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives for grouped expert computation, kernel fusion, and low-precision training. As biological foundation models grow in complexity, TE, along with techniques like the MXFP8 format and GroupedMLP kernel, offers a pathway to expand model capacity while improving GPU efficiency. NVIDIA’s Transformer Engine (TE) implements…