QuickReduce INT3 Quantization and Benchmarking on MI355

8 min read. | 2058 total words.

Large Language Models (LLMs) typically contain billions — or even tens of billions — of parameters. During inference, tensor parallelism (TP) is a widely used technique that distributes the compute across multiple GPUs. This approach, however, requires frequent, large-scale data synchronization between layers, introducing significant communication latency and placing enormous pressure on interconnect bandwidth.

Among the various communication patterns, all-reduce stands out as one of the most critical. It aggregates data from all participating devices (for example, via summation) and broadcasts the result back to each device, enabling synchronized multi-GPU computation.

QuickReduce is a high-performance all-reduce library designed for AMD ROCm that supports inline compression: the send path quantizes the data, the reduction is performed in the compressed domain, and the receive path dequantizes the result. Compared to RCCL (AMD's collective communication primitives for multi-GPU and multi-node communication), QuickReduce achieves up to 2.25x speedup on 2×MI300X and 4×MI300X configurations, and outperforms RCCL across all multi-GPU (single-node) configurations when optimized, as reported in our previous work.

In our previous work, we discussed the design principles and performance characteristics of QuickReduce in detail, integrated it into the popular inference frameworks vLLM and SGLang, and provided comprehensive performance and accuracy benchmarks for integer quantization (INT8/INT6/INT4) on MI300.

In a subsequent post, we further introduced a hardware-accelerated FP4 quantization path on MI355 and presented updated performance and accuracy evaluations, showing that on MI355 FP4 and INT4 are essentially interchangeable.

In this blog post, we go one step further and implement the INT3 quantization mode of QuickReduce. INT3 trades a higher compression ratio for a smaller on-wire transfer volume. We describe its design and vLLM integration, and demonstrate its latency advantage over INT4 and its model-dependent accuracy on AMD Instinct MI355.

Each quantization block covers 32 FP16/BF16 values (8 threads × 4 fp16x2 lanes) that share one FP16 scale. Values are scaled, clipped to the symmetric range [-4, +3], rounded to integers, and biased to the unsigned domain [0, 7] for packing.

Each group of 32 values shares one decoding scale, written by the thread-group leader.

This packing shrinks the per-rank transmitted tile from 1152 bytes (INT4) to 896 bytes, a reduction of about 22%. The smaller on-wire data directly reduces the communication volume of the two-shot QuickReduce kernel in Phase-1/Phase-2; this benefit only takes effect once the message length exceeds the vLLM activation threshold.

The send path is implemented in CodecQ3::send (csrc/quickreduce/quick_reduce_impl.cuh): it first computes the absolute maximum within the thread group and derives the FP16 encode/decode scale; it then scales and clips the values to [-4, +3], rounds them, and biases them to [0, 7]; finally it splits each 3-bit value into q2w and q1w, writes them into the communication buffer, and the group leader writes the block scale.

The receive path is implemented in CodecQ3::recv: it reads q2w, q1w, and the scale from the buffer using non-temporal loads, recombines the low 2 bits and high 1 bit into an unsigned quantized value, maps it back to the symmetric integer domain, and multiplies by the decoding scale to reconstruct FP16/BF16. The two-shot kernel calls recv during Phase-1B (in-segment reduction) and Phase-2 (gather), consistent with the other QuickReduce quantization modes.

All experiments in this post were run on a single-node MI355 server (8× AMD Instinct MI355X GPUs in one node), so every TP configuration below uses GPUs within that single node (for the full software and system configuration, see the Appendix). On this server we benchmark three all-reduce implementations:

Custom AllReduce (CR) — A widely adopted all-reduce acceleration kernel used in vLLM and SGLang, which significantly outperforms RCCL at low data volumes. QuickReduce (QR) — This post focuses on comparing the INT3 and INT4 quantization configurations.

The tests cover message sizes ranging from 4 KB to 1 GB, and all latency values are reported in μs (microseconds).

To better visualize relative performance across the full message-size range, we plot speedup over RCCL rather than raw latency. The y-axis is defined as RCCL latency ÷ each method's latency, so a value greater than 1 indicates better performance than RCCL.

The figure below shows how the TP=2 speedup over RCCL varies with message size, highlighting where QuickReduce INT3 and INT4 overtake CR. Speedup over RCCL on MI355 at TP=2. The y-axis is computed as RCCL latency divided by the measured latency of each method.#

Based on our kernel experiments, INT3 and INT4 perform very similarly under TP=4 and TP=8. We believe that as TP grows, the number of GPUs that must communicate with one another increases multiplicatively, requiring more synchronization and scheduling time; in this regime, synchronization and latency overheads — rather than the transferred data volume — increasingly become the bottleneck, so the transfer time saved by further bit-width compression (INT3) becomes a smaller fraction of the total and the benefit is diluted.

Because INT3 compresses the bit-width more aggressively and is more sensitive to accuracy, taking both performance and accuracy into account, we do not recommend using INT3 outside of TP=2.

For message sizes beyond the crossover point, QuickReduce delivers the highest large-message speedup over RCCL: at a 1 GB message size, QR INT3 reaches roughly 5x speedup at TP=2. Compared to INT4, INT3 only shows a clear additional speedup at large communication volumes (above ~4 MB); for smaller messages the two are essentially on par.