NVIDIA MoE models activate just 3 billion parameters in a 30B system

Source: Medium · developer.nvidia.com Nemotron 3.5 Lightning illustrates that large language models do not necessarily require activating all parameters with each processing step; the 30 billion-parameter model functions by activating just 3 billion parameters per token. This is achieved through a Mixture-of-Experts (MoE) architecture, which selectively routes incoming text to a subset of its networks, the company says. “How a model organizes its parameters matters as much as how many it has,” highlighting that parameter organization impacts throughput and memory cost more than sheer parameter count. Like engines with varying cylinder activation, the approach prioritizes efficiency without sacrificing capacity. MoE Architecture Activates Fewer Parameters in Nemotron 3.5 Lightning Nemotron 3.5 Lightning activates only three billion parameters per token despite possessing a total of 30 billion parameters, a demonstration of efficient scaling within Mixture-of-Experts architectures. This selective activation contrasts sharply…