Quark Support for HuggingFace Diffusers and SVDQuant

10 min read | 2530 total words.

Diffusion models are heavy on memory and compute: a single text-to-image call runs a large transformer or UNet dozens of times. Quantization — storing weights (and sometimes activations) in low precision — is one of the most effective ways to cut both the memory footprint and the latency of these models.

In a previous post [1] we showed how AMD Quark [2] brings MXFP4 quantization to Diffusers [3] and xDiT FLUX.1-dev [4] image generation, reaching up to 1.92× speedup over the BF16 eager baseline on a single AMD Instinct™ MI350 GPU while preserving quality. This post covers two new capabilities for diffusion models: SVDQuant [5], for accurate low-bit (4-bit) quantization, and Quark's native Hugging Face Diffusers support. Quark now allows quantized diffusion models to be saved and reloaded through the standard Diffusers save_pretrained / from_pretrained APIs.

Pushing diffusion models to 4-bit — quantizing not just weights but activations too — is much harder than the FP8 or INT8 case. Both the weights and the activations in diffusion transformers contain outlier channels whose magnitudes dwarf the rest of the tensor. A naive 4-bit grid has to stretch to cover those outliers, which crushes the resolution available for the bulk of the values and shows up as visible artifacts in generated images.

Smoothing migrates part of the activation dynamic range into the weights (as in SmoothQuant [6]), so neither side has to absorb the full outlier magnitude alone.

A high-precision low-rank correction branch captures the residual outliers that low-bit quantization cannot represent. The weight matrix is decomposed via SVD; at inference, the output is the result of the low-bit residual GEMM plus a small rank-16-to-32 correction. That correction is cheap in compute and memory, yet recovers most of the accuracy lost at 4 bits.

This is what makes 4-bit activation quantization viable, not just weight-only compression: the w4a4, mxfp4, and nvfp4 modes below quantize activations as well as weights. In Quark, SVDQuant is configured through SVDQuantConfig, with ready-made schemes from build_quant_layer_config:

Native inference. On MI300 / MI350 GPUs, quark.torch.enable_native_inference runs the low-bit residual GEMM on AMD AITER [10] matrix-core kernels. For SVDQuant, the low-rank correction runs as a separate branch alongside that GEMM (optionally overlapped on a second CUDA stream via RuntimeOptions(svdquant_overlap_streams=True)); it is not yet fused into the GEMM kernel — a fused SVDQuant kernel is planned and will further reduce its latency. This path is part of AMD Quark's diffusion support.

The table below shows Quark quantization of FLUX.1-dev on a gfx950 / MI350 GPU, reported relative to the BF16 baseline:

Config (FLUX.1-dev)

Plain native FP4 runs about 1.12× faster than BF16 while cutting model memory by roughly a third. Native SVDQuant trades a little speed (currently ~0.92× of BF16, due to the extra correction branch that is not yet fused) for the accuracy that makes 4-bit activations viable, while still trimming model memory by ~27%. Once the correction branch is fused into the GEMM, SVDQuant's latency should move toward the plain FP4 path.

The images below are FLUX.1-dev generations produced with Quark SVDQuant across all four low-bit formats, using the same prompt and seed. Quality holds across INT4 and FP4:

Figure 1. FLUX.1-dev generated with Quark SVDQuant across four low-bit formats (INT4 weight-only, INT4 W4A4, MXFP4, NVFP4), same prompt and seed — image quality holds across all four.

These samples were produced with examples/torch/diffusers/testSVDQuant.py (multi-mode SVDQuant for SDXL / FLUX / SD3). On FLUX.1-dev, the SVDQuant W4A4, MXFP4, and NVFP4 variants all maintain CLIP scores [11] within about half a point of the FP16 reference:

SVDQuant (FLUX.1-dev)

CLIP score with openai/clip-vit-large-patch14 on 1,000 MJHQ [12] prompts (higher is better). "RTN" and "GPTQ" [13] denote how the 4-bit residual weights are rounded.

A single import quark.integrations.diffusers statement registers Quark as a Hugging Face Diffusers quantizer, so quantized diffusion models behave like any other Diffusers checkpoint. There are two ways to use it.

Offline quantization — quantize once, reload anywhere. Quantize a pipeline submodule with ModelQuantizer, save it with export_safetensors (which routes through save_pretrained and embeds the serialized Quark QConfig under quantization_config in config.json), and reload later with plain from_pretrained. The checkpoint is self-describing: the loader reads the config, rebuilds the quantized layers (with meta-device and low_cpu_mem_usage loading supported), loads the weights, and freezes the model for inference — no QConfig needed at the call site.

Native online quantization — quantize at load time. Pass quantization_config=... to from_pretrained against a plain fp16/bf16 checkpoint, and Quark applies weight-only quantization in-process, with no export/reload round-trip. We already offered online quantization through xDiT [1] — where Quark replaces a transformer's linear layers with FP8 or MXFP4 implementations at load time and routes them to AITER kernels — and that same online path is now available directly through Diffusers.

Availability. Today this is enabled by importing quark.integrations.diffusers, which self-registers (monkeypatches) the "quark" method into the Diffusers registries at runtime. We also have a PR to add Quark to Diffusers upstream (huggingface/diffusers#14077); until it merges, the one-line import is all that's needed — nothing else in your code changes.

This tutorial is adapted from the scripts in examples/torch/diffusers (quantize_diffusers.py, testSVDQuant.py). It uses SDXL [14] and FLUX.1-dev [4] as running examples; the same pattern applies to SD1.5 [15], SD3 [16], and PixArt [17] by swapping the pipeline class and the target submodule.

The setup mirrors the earlier xDiT blog. On an AMD Instinct GPU with a recent ROCm PyTorch image:

Which submodule do you quantize? For SDXL / SD1.5 it is pipe.unet; for FLUX / SD3 / PixArt it is pipe.transformer. Quantization is applied to that submodule, not the VAE or text encoders.

1. Quantize, Save, and Reload through Diffusers#

This is the headline workflow. Weight-only quantization needs no calibration data, so it is the simplest way to see the full round-trip: quantize the target submodule with ModelQuantizer, save it with export_safetensors (which routes through save_pretrained and embeds the Quark QConfig in config.json), and reload it later with plain from_pretrained.

Reloading is a two-liner. Importing the integration self-registers the "quark" method; from_pretrained then reads quantization_config, reconstructs the quantized layers via process_model_transformation, loads the checkpoint, and freezes the model for inference — no QConfig needed at the call site:

The same config.json mechanism works when the quantized submodule lives inside a full pipeline directory: DiffusionPipeline.from_pretrained("<org>/<checkpoint>") reloads the whole quantized pipeline in one call. FP8 and other standard PTQ schemes follow the identical export/reload flow (activation-quantized schemes additionally need the calib