Reproducing AMD MLPerf Training v6.0 Submission Result

This blog provides a step-by-step guide for reproducing AMD’s MLPerf Training 6.0 submission results. AMD submitted results on three benchmarks this round:

Reproducing AMD MLPerf Training v6.0 Submission Result#

8 min read. | 1841 total words.

This blog provides a step-by-step guide for reproducing AMD’s MLPerf Training 6.0 submission results. AMD submitted results on three benchmarks this round:

Llama 3.1 8B pretraining — full pretraining from random weights on a subset of the C4 dataset

Flux.1-schnell — text-to-image training benchmark on MI325X (8-node)

Results were submitted on AMD Instinct MI325X, MI350X and MI355X GPUs. For the first time, AMD’s Primus training framework was used in all LLM submissions (Llama 2 70B LoRA fine-tuning and Llama 3.1 8B pretraining). This guide covers the reproduction procedure for each benchmark, including environment setup, dataset preparation, training configuration, execution, and result validation.

Readers looking for context on the performance results and competitive analysis should refer to the companion blog post: Technical Dive into AMD’s MLPerf Training v6.0 Submission.

ROCm 7.2.2 or later

Slurm (for the multinode Flux.1-schnell training submission)

At least 6 TB of disk space (for Flux.1-schnell dataset preparation)

See the ROCm Quick start installation guide for information on how to install ROCm.

In the following sections, you will first find instructions to set up the environment and preprocess the dataset for training. Then, you will go through the steps to run the model training job and collect benchmark results. Finally, you will be presented with the benchmark results submitted to the MLPerf Training v6.0 round by AMD.

The LLM benchmarks submitted by AMD in this round are powered by Primus, AMD’s unified, modular training framework for large-scale foundation model training on AMD Instinct GPUs. Primus abstracts over multiple training backends — Megatron-LM and TorchTitan — through a single CLI and configuration system, enabling pretraining, fine-tuning, and RLHF workflows without managing each backend separately.

Primus (Primus-LM): The unified training framework. Provides the primus-cli, configuration templates, workflow orchestration, and ROCm-native integrations with Megatron-LM and TorchTitan.

Primus-Turbo: A high-performance operator library delivering optimized FlashAttention, GEMM, and communication kernels for Instinct GPUs via AITER, Composable Kernel, and ROCm Triton — activated as a non-intrusive drop-in over the training framework.

Primus-SaFE: A cluster management and fault tolerance layer for enterprise-grade training resilience at scale.

All three are bundled in the Docker images for all benchmarks. No additional installation or configuration is required — Primus is activated automatically by the benchmark configuration scripts.

Users interested in using Primus for their own workloads can get started at github.com/AMD-AGI/Primus. Pre-built Docker images are available from the rocm/primus registry on Docker Hub.

The Llama 2 70B LoRA fine-tuning benchmark measures how quickly a model can be fine-tuned to reach a target ROUGE-2 score on the GovReport summarization dataset, using Low-Rank Adaptation (LoRA) with FP8 precision.

Convergence target: ROUGE-2 score of 0.925

Docker image: rocm/amd-mlperf:llama2_70b_training_6.0

GovReport is a dataset for long document summarization that consists of reports written by U.S. government research agencies. The dataset hosted on the MLPerf drive is already tokenized and packed so that each sequence has a length of 8,192.

The model used in this submission is the Llama 2 70B with fused QKV. You will need 270 GB of disk space to download and convert the model.

To download the model from Huggingface, you will need to sign the LLAMA 2 COMMUNITY LICENSE AGREEMENT as well as obtain a Huggingface Token (HF_TOKEN).

Start the docker container by mounting the volume you want to use for downloading the data under /data within the container. In this example we use /data/mlperf_llama2 as the host download directory:

Set the directory for the data, model and results. Ensure that $LOGDIR has write access for the results to be written by running sudo chmod -R 777 $LOGDIR. In this example we use /data/mlperf_llama2/results as the results directory, so please make sure to create this directory

config_MI350X_1x8x1.sh

config_MI355X_1x8x1.sh

For example, to set the configuration for the MI355X platform on a single node, use the config file config_MI355X_1x8x1.sh:

To perform a single run for Llama 2 70B LoRA fine-tuning, set NEXP to 1, and then run the run_with_docker.sh script:

To prepare for a 10 run result, simply set the environment variable NEXP to 10 before running the run_with_docker.sh script:

After the benchmarking is completed, the logs will be available under the directory $LOGDIR, where you can also find the benchmark results.

Note: To optimize the machine’s performance, the training script will also execute runtime_tunables.sh script before any training run.

Llama 3.1 8B Pretraining#

AMD led the development of the Llama 3.1 8B pretraining benchmark in MLPerf Training. Unlike Llama 2 70B LoRA, this benchmark starts from random weights — no checkpoint conversion is required. It trains on a subset of the C4 (Colossal Cleaned Common Crawl) dataset and targets a validation loss perplexity of 3.3.

Convergence target: Validation loss perplexity ≤ 3.3

Docker image: rocm/amd-mlperf:llama31_8b_training_6.0

Setup Docker Image - Llama 3.1 8B#

Prepare Dataset and Model - Llama 3.1 8B#

The dataset used for this benchmark according to MLCommons is the c4/en/3.0.1 dataset from AllenAI.

The pre-tokenized dataset and the tokenizer are available for download. You can navigate to your desired download directory and run the following commands to download the dataset and tokenizer. In this example, the folder /data/mlperf_llama31_8b is used as the download directory on the host.

After the download is completed, you should see files with the following naming conventions under the data directory, ending with both .idx and .bin extensions:

Training partitions: c4-train.en_6_text_document

Validation partitions: c4-validation-91205-samples.en_text_document

The size of the data directory is ~80 GB, and ~30 GB for the model directory.

Run Pre-training Benchmark - Llama 3.1 8B#

Setup Environment - Llama 3.1 8B#

Set the directory for the data, model and results. Ensure that $LOGDIR has write access for the results to be written by running sudo chmod -R 777 $LOGDIR, In this example the folder /data/mlperf_llama31_8b/results is used as the results directory, so please make sure to create this directory.

Set Configurations - Llama 3.1 8B#

Similar to the Llama 2 70B LoRA fine-tuning case, there is a configuration file for each Instinct platform submission for the Llama 3.1 8B benchmark:

config_MI350X_1x8x1.sh

config_MI355X_1x8x1.sh

Launch a Single Training Run - Llama 3.1 8B#

Launch 10 Training Runs - Llama 3.1 8B#

After completion, the logs will be available under the directory $LOGDIR.

To optimize the machine’s performance, the training script will also execute the runtime_tunables.sh script before any training run.

Flux.1-schnell Text-to-Image Training#

This benchmark trains the Flux.1-schnell model using preprocessed CC12M training data and COCO validation data. See the MLCommons benchmark specification for quality rules and dataset details. AMD’s v6.0 submission for this benchmark targets the MI325X platform in an 8-node × 8-GPU configuration (config_MI325X_08x08x32.sh) coordinated via Slurm.

At least 6TB disk space is required.

GPUs are not required for dataset preparation.

Set up Docker Image - Flux.1#

Make sure the image is accessible on every node of the run environment.

Prepare Dataset - Flux.1#

The dataset download and preprocessing scripts are included in the container. GPUs are not required for dataset preparation. In this example, /data/mlperf_flux1/data is used as the host download directory. The dataset path must be on a shared filesystem reachable from every node in the Slurm job.

Optionally, reclaim disk space by deleting /dataset/cc12m_preprocessed and /dataset/coco_preprocessed after conversion.

All training commands below should be run from the ./code directory on the host. Edit job.sh in ./code to set Slurm options, partition, and the variables below. Set these inside job.sh (do not rely on exporting them in your shell before sbatch).

rocm/amd-mlperf:flux1_training_6.0

Example job.sh excerpt:

Ensure $LOGDIR exists and is writable on the shared filesystem (mkdir -p "$LOGDIR" && sudo chmod -R 777 "$LOGDIR").

The script run_with_docker_slurm.sh launches one container per node, sources config_MI325X_08x08x32.sh (selected by DGXSYSTEM), sets up RCCL/torch distributed across the cluster, and runs the training loop. Per-run logs land under $LOGDIR on the shared filesystem.

To optimize each node’s performance, the training script also executes runtime_tunables.sh on every node before each training run.

Use the same run_start / run_stop timestamps for time-to-train. Convergence is validation loss at or below 0.586 (see Evaluation above). For a submission score, average the 8 of 10 runs after dropping the fastest and slowest.

Below is the log from one of the training runs on the AMD MI355X platform for the Llama 3.1 8B pretraining benchmark:

From the logs, the run_start and run_stop events happened at timestamp 1778651870949 and 1778657049089 in milliseconds respectively. The difference between these two timestamps is the time taken for the training to finish, and is equal to 86.3 min. In most cases, this should serve as a reliable estimate of the MLPerf Training score, but obtaining an MLPerf compliant score is more involved, as described below.

Time-to-train is determined by two components: throughput and number of processed samples. Throughput is a measure of hardware performance and from the log above, it is about 30.7 samples per second for the Llama 3.1 8B pretraining benchmark running on MI355X. You can find throughput printed in your output file by searching for the last occurrence of throughput in the file. If your throughput is significantly lower, this indicates a misconfiguration or hardware issue that needs to be addressed.

Number of processed samples, given by samples_count in the output file, is determined by how fast the training converges to the defined accuracy target. In MLPerf Training, convergence needs to match the Reference Convergence Checkpoints (RCPs). For hyperparameters used in the AMD submission for an individual run of the Llama 3.1 8B pretraining benchmark on MI355X, a value of 184,320 as in the log shown above is most common.

To obtain an MLPerf compliant Time-to-Train score, you will need 10 consecutive runs. Calculate timings for each run using the run_start and run_stop timestamps as described above. Disregard the slowest and fastest runs and average the remaining 8 runtimes. Next, check convergence of runs by running RCP Checker Script on the directory containing the set of 10 runs. The output should look like this:

Make sure there is no error in the output. If RCP checker prints a normalization factor, the average of 8 runs should be multiplied by the factor to arrive at the final score.

For MI355X, the score should be about 8.5 mins. The AMD MLPerf Training v6.0 submission score is 8.27 mins (Submission ID 6.0-0036).

For MI350X, the score should be about 10 mins. The AMD MLPerf Training v6.0 submission score is 10.25 mins (Submission ID 6.0-0035).

Llama 3.1 8B Pretraining Results#

For MI355X, the score should be about 87 mins. The AMD MLPerf Training v6.0 submission score is 86.84 mins (Submission ID 6.0-0036).

For MI350X, the score should be about 110 mins. The AMD MLPerf Training v6.0 submission score is 109.76 mins (Submission ID 6.0-0035).

Flux.1-schnell Text-to-Image Training Results#

For MI325X (8 nodes × 8 GPUs), the score should be about 92.5 mins. The AMD MLPerf Training v6.0 submission score is 92.36 mins (Submission ID 6.0-0037).

This guide covered the full reproduction workflow for AMD’s MLPerf Training v6.0 submission across three benchmarks — Llama 2 70B LoRA fine-tuning, Llama 3.1 8B pretraining, and Flux.1-schnell text-to-image training — on AMD Instinct MI325X, MI350X and MI355X GPUs. You can view the official MLPerf Training v6.0 results on MLCommons. Keep in mind that slight variations in hardware setups and conditions may lead to minor discrepancies from AMD’s results. Primus, AMD’s unified training framework for large-scale foundation model training, powers all benchmarks and is bundled in the respective Docker images with no additional setup required. For more insights into the techniques used to optimize training workloads in these submissions, check out this blog. Ready to go further? Build on this foundation to scale your AI workloads with AMD Instinct™ GPUs and ROCm.

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, ROCm, Instinct, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved

analiza aiqbench

Reproducing AMD MLPerf Training v6.0 Submission Result

Subiecte conexe