Inference is Everywhere…Even the Fabric Switch: Why AI Needs In-Network Compute Today

In-Network Compute (INC) is the execution of collective operations at network switches. INC is most impactful for inference workloads, where performance, latency, and efficiency directly shape user experience and deployment cost. Scale-up is the best INC use case because the most demanding inference systems depend on tightly coupled, high-bandwidth, low-latency communication within the node or pod. Four factors must be accounted for before INC is widely adopted across the AI industry. Scorpio™ X-Series Smart Fabric Switches bring INC to open scale-up fabrics, improving tokens per watt performance. As AI model sizes have grown to trillions of parameters and deployment has shifted from training to production inference, two trends are driving urgency for the network switch fabric to become the major driver of AI system performance. Inference workloads are surpassing training workloads in total volume1, and operations essential to completing them sit on the unavoidable critical execution path through the switch. Even microseconds of latency in these operations translates directly to token latency for the…