aiqbench analysis
In-Network Compute (INC) addresses the growing latency bottleneck in AI inference by offloading collective operations like AllReduce directly onto network switches. This architecture is particularly effective for small message sizes typical of modern tensor parallelism, allowing data aggregation and distribution to occur without consuming accelerator compute cycles. As inference workloads surpass training in volume, integrating INC into scale-up fabrics becomes critical for maintaining throughput in large-scale clusters.
AllReduce Speedup With In-Network Compute: How INC Accelerates AI Collective Operations
In-Network Compute speeds up collective operations for the data transfers that drive AI inference today by improving communication latency. The post AllReduce Speedup With In-Network Compute: How INC Accelerates AI Collective Operations appeared first on ASTERA LABS, INC..
INC reduces the transfer time component of latency for collective operations. The tensor parallelism that drives modern inference means INC is most impactful at small message sizes. AllReduce is an increasing percentage of inference time as cluster sizes grow. The inflection point facing AI today as inference workloads overtake training workloads in volume1 has created an industry-wide need for the execution of collective operations within network switches, or In-Network Compute (INC). Today’s inference workloads utilize massive models distributed across scale-up fabrics comprising numerous accelerators, and as a result rely heavily on operations like AllReduce to orchestrate the matrix multiplications that make them possible across this fabric. ReduceScatter: Aggregation of data is performed across all involved accelerators. A single “slice” of total data is reduced and stored in each GPU. AllGather: Local copies of the scattered slices…
Excerpt shown under the publisher's RSS terms · 136 words · full text and figures are on the publisher's site
Related coverage
Inference is Everywhere…Even the Fabric Switch: Why AI Needs In-Network Compute Today
Astera Labs · Nov 5, 2026
Collective Strong Quantum Mpemba Effect in Accelerated Quantum Detectors
arXiv · Oct 20, 2026
Duet: Dual-Robot Understanding via Efficient Teaching
arXiv · Sep 24, 2026
TILDE: Teacher-in-the-Loop Distillation from Embodied Foundation Models
arXiv · Oct 21, 2026