Technical Dive into AMD MLPerf Inference v6.1 Submission

Technical Dive into AMD MLPerf Inference v6.1 Submission#

September 17, 2026 by Meena Arunachalam, Miro Hodak, Poovaiah Palangappa, Uma Kannikanti, Rita Brugarolas, Hemanth Acharya, Karan Verma, Yamini Preethi Kamisetty, Rebecca Lee, Neha Matthews, Rajesh Poornachandran, Karan Verma, Jiawei Chen, Huasha Zhao, Mikko Lauri, Jesus Carabano Bravo, Nico Holmberg, Eliot Li.

15 min read. | 3759 total words.

MLPerf Inference v6.1 results were released on September 16, 2026. For AMD, this was a highly successful round in which we achieved several leadership scores, expanded benchmark coverage, introduced a new GPU with validated performance, and enabled record-setting results from our partners. In this round, AMD and its partners provided validated results across a diverse set of workloads, including dlrm-v3 (Deep Learning Recommendation Model v3), llama2-70b (Meta's 70-billion-parameter language model), gpt-oss-120b (an open-weight 120-billion-parameter Mixture-of-Experts model), deepseek-r1 (DeepseekAI's 671-billion-parameter Mixture-of-Experts model), and Wan 2.2-t2v (a 14-billion-parameter text-to-video generation model). Single-node results used 8 GPUs per node; gpt-oss-120b was additionally submitted at multi-node scale across 72 AMD Instinct™ MI355X GPUs. All results were validated through MLPerf peer review.

This blog provides an overview of the optimizations used to achieve the results and walks readers through the results we achieved. The results are fully reproducible and readers can do their own measurements by following step-by-step instructions in our reproduction blog.

Leadership performance: AMD Instinct™ MI355X and MI350P GPUs delivered leading results across multiple benchmarks, establishing AMD Instinct as a top-tier platform for today's most demanding AI workloads.

Leadership performance: AMD Instinct™ MI355X and MI350P GPUs delivered leading results across multiple benchmarks, establishing AMD Instinct as a top-tier platform for today's most demanding AI workloads.

A powerful new accelerator: This round introduces the AMD Instinct™ MI350P, a PCIe-based GPU for enterprise inference that posted the top result among all PCIe-based submissions.

A powerful new accelerator: This round introduces the AMD Instinct™ MI350P, a PCIe-based GPU for enterprise inference that posted the top result among all PCIe-based submissions.

Broader benchmark coverage: AMD expanded its footprint over prior rounds, spanning recommendation, multimodal, reasoning, and large-language-model inference - giving customers validated performance across a wider range of workloads than ever before.

Broader benchmark coverage: AMD expanded its footprint over prior rounds, spanning recommendation, multimodal, reasoning, and large-language-model inference - giving customers validated performance across a wider range of workloads than ever before.

Record-setting scale: In collaboration with our partners, AMD powered the largest-scale submission in MLPerf Inference history, a 512-GPU configuration, showcasing how AMD Instinct GPUs scale in real-world deployments.

Record-setting scale: In collaboration with our partners, AMD powered the largest-scale submission in MLPerf Inference history, a 512-GPU configuration, showcasing how AMD Instinct GPUs scale in real-world deployments.

Record-breaking throughput: The same collaboration delivered the highest total LLM throughput ever recorded in an MLPerf submission, underscoring the raw performance of AMD Instinct GPUs at scale.

Record-breaking throughput: The same collaboration delivered the highest total LLM throughput ever recorded in an MLPerf submission, underscoring the raw performance of AMD Instinct GPUs at scale.

A thriving partner ecosystem: Seven partners submitted results across multiple AMD Instinct GPUs that closely tracked results by AMD - proving these results are reproducible from a single server to the largest clusters.

A thriving partner ecosystem: Seven partners submitted results across multiple AMD Instinct GPUs that closely tracked results by AMD - proving these results are reproducible from a single server to the largest clusters.

Fifth MLPerf appearance: Fifth consecutive MLPerf Inference for AMD round reflects our sustained investment in transparent, standards-based benchmarking.

Fifth MLPerf appearance: Fifth consecutive MLPerf Inference for AMD round reflects our sustained investment in transparent, standards-based benchmarking.

This round introduces the AMD Instinct™ MI350P PCIe® GPU, a new addition to the AMD Instinct portfolio built on the CDNA™ 4 architecture. PCIe-based and purpose-built for enterprise generative and agentic AI, its standard full-height, full-length dual-slot form factor drops into mainstream air-cooled servers — with no specialized cooling or rack redesigns — and supports up to eight cards per server. Each card offers 144 GB of HBM3E, 4 TB/s of memory bandwidth, and a configurable 450–600W TDP, and delivers a peak of 4600 TFLOPS at the MXFP4 precision format well-suited to modern AI inference[1]. Because it shares the CDNA 4 architecture with the rest of the AMD Instinct MI350 series, the MI350P GPU readily benefits from the computational optimizations developed across those GPUs. Backed by the open-source AMD Enterprise AI stack, it brings datacenter-class AI performance into existing infrastructure and power envelopes with favorable total cost of ownership.

To achieve the highest inference performance, we used the AMD Instinct™ MI355X GPU. Designed specifically for AI workloads, high-performance computing (HPC), and large-scale cloud deployments, AMD Instinct MI355X GPU leverages AMD advanced CDNA4 architecture to deliver exceptional compute power and efficiency. The AMD Instinct MI350 Series GPU specifications are shown below.

Figure 2: AMD Instinct MI350 Series specifications.#

Figure 2: AMD Instinct MI350 Series specifications.#

Overview of MLPerf Inference v6.1 Benchmark Tasks#

AMD participated in the closed division, datacenter category and submitted results for five models across multiple benchmark configurations. The closed division enforces a strict accuracy floor — 99% of the reference score for most workloads, and 99.9% for dlrm-v3 — and requires documented, reproducible system configurations. The table below lists each submitted workload:

99.9% of FP32 reference

99.9% of FP32 reference

MoE layers account for most of the compute in gpt-oss-120b. AMD optimized the FlyDSL MoE a8w4 kernels — 8-bit activations against 4-bit weights — and tuned them for the token distributions each scenario actually produces, yielding an 8% end-to-end improvement in Offline and a 5% improvement in the Server scenario.

AMD replaced the single unified attention kernel with separate kernels for the prefill and decode phases, improving attention performance by 25–28%. Prefill is a compute-bound problem over a contiguous query block, while decode is memory-bound over a paged KV cache, so one kernel serving both could only ever be a compromise for each. Prefills are now served by AITER's variable-length flash attention, which handles causal masking, the sliding window, and attention sinks inside the kernel. Decode is served by a new paged-attention kernel written in Gluon.

Rewriting the decode kernel in Gluon gave explicit control over MFMA and shared-memory layouts and over instruction scheduling, which is what made the memory pipeline changes possible: 16-byte coalesced loads over a shuffled KV cache layout, two K buffers kept live so each block's load is issued underneath the previous block's P·V MFMA, and V streamed underneath the QK MFMA. With an FP8 KV cache, both matrix multiplications run in FP8 on the native 16x16x32 MFMA instructions of the CDNA 4 architecture. Sliding-window layers collapse to a single context partition with attention sinks folded inline into the softmax denominator, removing the reduction kernel and its scratch traffic.