Efficiently Serving NVFP4 Models on AMD Instinct™ MI350X/MI355X Accelerators via Online NVFP4 to Quark MXFP4 Requantization

15 min read. | 3757 total words.

A growing share of frontier open-weight models are available in NVFP4 format, but on some AMD GPUs (MI350X / MI355X), FP4 compute support is limited to MXFP4. This means NVFP4 checkpoints couldn't run on the native 4-bit acceleration path these GPUs provide.

To close that gap, we present an online NVFP4 to MXFP4 requantization pipeline in SGLang that converts NVFP4 weights to MXFP4 once at weight loading time and serves them through the AMD GPU's native MXFP4 path. In this blog you will explore how the pipeline is designed, see how its serving throughput and accuracy stack up against native MXFP4 checkpoints and an emulated NVFP4 reference, and learn how to serve your own NVFP4 checkpoints directly on MI350X / MI355X GPUs, with three key benefits:

Production-ready, high-performance MXFP4 inference pipeline is used, enabling serving throughput on par with that of the native MXFP4 checkpoints on MI350X / MI355X GPUs.

NVFP4 is a popular 4-bit floating-point format (packed E2M1 with an FP8 per-block scale and an FP32 per-tensor scale). Checkpoints like nvidia/Qwen3.5-397B-A17B-NVFP4 and nvidia/Kimi-K2.6-NVFP4 fit very large models into a fraction of the memory their bf16 versions would need.

However, there is a catch for AMD GPU users: GPUs like MI350X and MI355X implement their native 4-bit compute pipeline in MXFP4, not NVFP4. The two formats both pack two FP4 (E2M1) values per byte, but they differ in the following important aspects:

Because the block sizes and scale encodings differ, an NVFP4 tensor can't be fed directly into an MXFP4 serving backend. Historically, this left users with two options: run an offline conversion tool and manage a second copy of every NVFP4 checkpoint, or emulate NVFP4 by dequantizing the weights to bf16 and running standard bf16 GEMM (slow and impractical for large scale inference with the largest models).

In this blog, we present a third option that avoids both downsides: an online NVFP4 to MXFP4 requantization pipeline integrated in SGLang that enables efficient NVFP4 checkpoint serving on AMD GPUs like the MI350X and MI355X. The rest of this post walks through the pipeline's design, performance and accuracy comparison, and some usage guidance.

When serving in SGLang using the --quantization quark_mxfp4 flag, SGLang will automatically detect the NVFP4 source format, stream each weight tensor through a requantization step, and hand the resulting MXFP4 weights to the native MXFP4 compute kernels. Layers the checkpoint producer chose to keep in higher precision, like FP8 layers in mixed-precision checkpoints, will branch off to their own load-as-is paths (Figure 1).

Figure 1. The online requantization pipeline.#

Note that the requantization step only runs once during model load on a per-layer basis, so there is no per-request conversion overhead, and at steady state the serving throughput of this pipeline is indistinguishable from that of a natively MXFP4-quantized one.

A naive requantization approach that dequantizes the whole model to bf16 at once before requantizing would need hundreds of gigabytes of disk scratch space and memory for a frontier model. Instead, with the online NVFP4 to MXFP4 requantization feature in SGLang, we requantize one layer at a time as its weights load: materializing just that layer's dequantized intermediate, quantizing it to MXFP4, writing the result straight into a preallocated destination, and freeing the intermediate before moving on. Peak extra memory in the pipeline is bounded by a single layer, not the entire model.

Both ModelOpt and AMD Quark NVFP4 exports are recognized: Source checkpoint metadata is read from either config.json or a standalone hf_quant_config.json.

Producer excludes honored: Modules deliberately left in higher precision (listed under ignore / exclude_modules / exclude) are automatically kept out of the requantization step, so a layer meant to stay in high precision never gets silently quantized.

Mixed-precision support: Checkpoints like nvidia/Qwen3.5-397B-A17B-NVFP4-V2 ("quant_algo": "MIXED_PRECISION") utilize different precision formats per layer. This is also automatically recognized and supported.

At serve time, activations are quantized to MXFP4 dynamically for each batch rather than from stored scales. This means no calibration pass and no pre-computed activation scales are needed, and both operands of every GEMM are in MXFP4, so the matmul runs directly on the GPU's native 4-bit hardware acceleration path.

Performance Comparison of Online NVFP4 to MXFP4 Requant vs. Native AMD MXFP4 Checkpoints#

Performance is always one of the most important questions when it comes to online requantization. Does the online requantization path give up any performance compared to a checkpoint that was quantized to MXFP4 offline and shipped as-is?

AMD publishes native MXFP4 checkpoints for many popular models, so we perform a direct serving throughput comparison: nvidia NVFP4 checkpoints with --quantization quark_mxfp4 (online NVFP4 to MXFP4 requant) vs. an AMD offline quantized MXFP4 checkpoint with the same quantization configuration on an identical serving stack and workload. All results presented below are from a steady state (warmed-up, averaged across 10 runs). As shown in Figure 2 below, the two paths track each other closely across all five model checkpoints tested.

Figure 2. Steady-state output-token throughput of the online NVFP4 to MXFP4 requant path vs. native AMD MXFP4 checkpoints across five models.#

Qwen3.5-397B-A17B: 14.28 / 14.18 ms Qwen3.5-397B-A17B-V2: 17.23 / 17.19 ms [Other models]: 17.30 / 17.66 ms, 18.90 / 19.14 ms, 25.59 / 25.09 ms

The benchmarked online NVFP4 to MXFP4 requant path throughput is very similar to that of the Native AMD MXFP4 path, with small deltas within run-to-run noise. The online path shows no significant steady-state throughput difference for converting at load instead of shipping a pre-quantized checkpoint. Commands and configuration to reproduce these throughput results are available in Appendix C.

The high serving throughput that the online NVFP4 to MXFP4 requantization feature enables is all thanks to the underlying MXFP4 serving path that's designed specifically for the CDNA4 hardware's MXFP4 compute capabilities. Several optimizations in SGLang's AMD MXFP4 path work together to improve FP4 serving throughput on AMD GPUs:

Native FP4 matmul: Both the dense linear layers and the MoE experts dispatch to AMD's AITER MXFP4 GEMM and fused-MoE kernels, which target the CDNA4 scaled matrix-core (MFMA) instructions directly. These units consume the OCP MXFP4 microscaling format natively, so the 4-bit operands feed straight into the tensor cores with no separate intermediate dequantize step.

Fusing activation quantization + GEMM: A W4A4 layer needs its activations in MXFP4 as well as its weights. For sma