Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark

Reinforcement learning (RL) for large language models (LLMs) alternates between two phases: generation (rollout), where the current policy produces responses, and training, where those responses are used to update the policy. In verl, the key design choices are when these phases run relative to each other (synchronously or with overlap) and where they run (colocated on the same GPUs or on separate GPU pools). This blog first explains the differences between the two modes and when to use each.

August 18, 2026 by Rishi Madduri, Tiffany Mintz, Mingjie Lu, Fuwei Yang, Vicky Tsang, Eric Kim, Yuankai Chen, Mukhil Azhagan Mallaiyan Sathiaseelan, Yao Liu, Jared Bowden, Joyce Zhang, Phani Vaddadi, Yao Fu, Vish Vadlamani.

13 min read. | 3138 total words.

Reinforcement learning (RL) for large language models (LLMs) alternates between two phases: generation (rollout), where the current policy produces responses, and training, where those responses are used to update the policy. In verl, the key design choices are when these phases run relative to each other (synchronously or with overlap) and where they run (colocated on the same GPUs or on separate GPU pools). This blog first explains the differences between the two modes and when to use each.

It then provides a hands-on walkthrough of running verl’s two fully asynchronous examples end-to-end on AMD Instinct™ MI355X GPUs with ROCm. You begin with Group Relative Policy Optimization (GRPO) on Qwen2.5-VL-7B-Instruct with the Geometry3k vision-math dataset and the Megatron trainer, and then run Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) on Qwen2.5-Math-7B with the DAPO-Math-17k and AIME-2024 datasets and the FSDP2 trainer. Finally, it presents throughput numbers for the synchronous use case on AMD Instinct™ MI355X compared to the NVIDIA B300.

These are two distinct studies: the async walkthroughs characterize pipeline behavior and GPU utilization on AMD Instinct™ MI355X using log-derived idle ratios and corresponding GPU busy-time, and the synchronous section separately compares MI355X and B300 throughput on a different workload.

For more background on verl on ROCm, see the earlier verl on ROCm blog.

Synchronous vs. Asynchronous RL: What They Are and Use Cases#

In a synchronous setup, the trainer does not start an update until a defined rollout unit, typically a full batch, has finished. Rollout and training still take turns, as they do not run in parallel on the same GPUs at the same time.

The most common synchronous mode in verl is colocated training with the hybrid engine, where training weights are offloaded, the vLLM rollout engine generates a batch, and then the system swaps back to run the optimizer step on the same GPUs. verl also supports disaggregated synchronous placement, where rollout and training use separate GPU pools but still follow a batch barrier, and the trainer waits for the full batch before updating.

Synchronous training keeps policy-version lag minimal. Samples are generated with weights that are current or very close to the weights being updated. That makes the loop easier to reason about and usually more stable, with fewer knobs for staleness and off-policy correction. The tradeoff is efficiency. Response lengths are often highly variable, so the batch cannot proceed until the slowest sample finishes. During rollout, workers that finish early can sit idle waiting on stragglers, and during training, no new rollouts are produced. On colocated setups, there is also overhead from switching between training and inference.

In an asynchronous setup, rollout and training overlap in time, usually on disaggregated GPU pools. While the trainer updates weights on its GPUs, the rollout fleet continues generating samples on separate GPUs. Weights are broadcast periodically (for example, via RCCL), and the trainer consumes samples from a buffer rather than waiting for one synchronized batch to complete.

One-step-off async overlaps training on batch t with generation of batch t+1, with roughly one step of policy lag. See the One Step Off Async Trainer documentation.

Fully Async Policy streams samples one at a time, with tunable staleness (staleness_threshold), sync frequency (trigger_parameter_sync_step), and partial rollout across weight updates. This is what the walkthroughs below use. See the Fully Async Policy documentation.

Async training reduces long-tail stalls and keeps training GPUs busy, especially when generation dominates wall-clock time. The cost is bounded off-policy staleness (samples may come from slightly older weights), plus extra orchestration: separate resource pools, weight sync, and often rollout correction or importance-sampling settings when lag grows.

Note: Disaggregated placement does not by itself mean asynchronous. verl’s fully async framework can also run in synchronous on-policy mode (for example, staleness_threshold=0). The async and sync distinction is about overlap, not only about separate fleets.

When to Use: Synchronous vs. Asynchronous?#

In reinforcement learning, generation length can be highly variable: many prompts within a single batch finish quickly, while a handful generate very long responses. Because the trainer must wait for the slowest sample in the batch, these long-tail generations stall the entire RL workflow. The training GPUs sit idle waiting on a few stragglers, and overall throughput is gated by the worst-case rollout rather than the average one.

verl’s Fully Async Policy trainer resolves this issue by physically separating the rollout and training fleets and letting them run at the same time. The following sections walk through both examples end-to-end: first the GRPO example on Qwen2.5-VL-7B-Instruct (Geometry3k vision-math, Megatron trainer), and then the DAPO example on Qwen2.5-Math-7B (DAPO-Math-17k and AIME-2024, FSDP2 trainer).

Qwen2.5-VL-7B-Instruct on the Geometry3k Vision-Math Dataset, Megatron Trainer#

This section walks through the GRPO example end-to-end, starting with what the Fully Async Policy trainer provides.

The fully asynchronous training system decouples the Trainer and the Rollouter and supports asynchronous sample generation and training. Because rollout and training own separate resources, they can be allocated more flexibly and support more flexible training logic. Concretely, the design provides:

Resource isolation: Unlike a hybrid engine, the Rollouter and Trainer use separate GPUs. Each fleet is sized independently.

Parallel generation and training: While the Trainer updates weights, the Rollouter is already generating the next samples.

Multi-step asynchronous: Compared to one step off policy, it supports asynchronous settings from 0.x steps to multiple steps, making the asynchronous solution more flexible.

RCCL parameter synchronization: Weight broadcasts from Trainer to Rollouter use the ROCm Collective Communication Library (RCCL) collective communication primitives.

Streaming inference and training: The Rollouter emits samples one at a time; a single sample is the minimum transmission unit.

Freshness control: async_training.staleness_threshold bounds how stale a sample can be and still be trained on.

Partial rollout: During parameter sync, in-flight rollouts are paused (sleep()) and resumed (resume()) rather than discarded, so you avoid paying the cost of regenerating them.

The supported training modes are Megatron and Fully Sharded Data Parallel (FSDP), with vLLM (server mode, AgentLoop) used for rollout.

This GRPO run is a demonstration to validate the async pipeline’s resource split, weight sync and partial rollout rather than model quality. The model is rewarded even as responses hit the truncation limit.

The following table summarizes the hardware and software configuration used for this run.

ROCm 7.0.2, PyTorch 2.9.1 (ROCm), vLLM, Megatron-core 0.16.0, TransformerEngine (ROCm)

rocm/verl:verl-0.7.1.amd0_rocm7.0.2_ubuntu22.04_py3.12_vllm0.20.2

Qwen2.5-VL-7B-Instruct (approx. 16 GB, 5 safetensors shards)

--device /dev/kfd --device /dev/dri + --group-add video expose the AMD GPUs to the container.

--shm-size=2048g gives Ray and the rollout workers plenty of shared memory for inter-process sample passing.

--ulimit memlock=-1 --ulimit stack=67108864 are required for RCCL/registered-memory pinning and deep stacks.

-w /workspace drops the user in the working directory where the verl repo already lives (/workspace/verl).

This confirms verl is imported from /workspace/verl/verl/__init__.py and all 8 MI355X (gfx950) GPUs are visible.

The example ships with a preparation script (prepare_geo3k_qwen25vl_7b_megatron_4_4.sh) for downloading the raw dataset, preprocessing it into parquet, and downloading the model. Inside the container run:

Dataset → ${HOME}/data/geo3k/{train,test}.parquet

Model → ${HOME}/models/Qwen2.5-VL-7B-Instruct

runs examples/data_preprocess/geo3k.py to emit parquet

hf downloads Qwen/Qwen2.5-VL-7B-Instruct. The relevant slice of the prepare.log:

The experiment runs /workspace/verl/verl/experimental/fully_async_policy/shell/geo3k_qwen25vl_7b_megatron_4_4.sh. It launches fully asynchronous GRPO for Qwen2.5-VL-7B on Geometry3k using the Megatron trainer config.

Resource split: this is the heart of “fully async”. With 8 GPUs, the script dedicates 4 to rollout and 4 to training:

algorithm.adv_estimator=grpo, use_kl_loss=True, kl_loss_coef=0.01, kl_loss_type=low_var_kl.

Trainer: Megatron with tensor_model_parallel_size=2, full CPU offload of params/grads/optimizer (param_offload, grad_offload, optimizer_offload, plus precision-aware optimizer offload): this is what lets a 7B VLM + Megatron fit comfortably on 4 GPUs.

Rollout: name=vllm, gpu_memory_utilization=0.8, max_model_len=32768, async server mode (VLLM_USE_V1=1, return_raw_chat=True).

Point the script at the model and launch (see Step 5). If you want the script to find the model automatically, you can instead export RAY_DATA_HOME=/root (it then resolves ${RAY_DATA_HOME}/models/Qwen2.5-VL-7B-Instruct).

Ray brings up the workers, and the four vLLM rollout engines initialize and capture HIP or CUDA graphs. The completion of the graph captures indicates the rollout is live.

A quick rocm-smi during the run is the clearest confirmation the async split is working: the four training GPUs sit near full utilization while the four rollout GPUs hold approx. 80% VRAM for the vLLM KV cache and burst as they generate:

The run completes cleanly after 50 trainer steps (200 global steps), in approx. 11 h, terminating on the total_rollout_steps budget:

Fully Async DAPO on Qwen2.5-Math-7B#

The same fully async machinery is not tied to one model, dataset, or trainer backend. To demonstrate this, the walkthrough runs a second example, dapo_7b_math_fsdp2_4_4.sh, which trains Qwen2.5-Math-7B on text-only math reasoning using the DAPO algorithm and an FSDP2 trainer (instead of Megatron). The Docker pull and launch steps are identical; only the preparation script, training script, and algorithm differ.

The verl repository implements DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization), a GRPO-family algorithm tuned for long-chain math reasoning. In this script it keeps GRPO’s group-relative advantage (adv_estimator=grpo) but layers on the DAPO-specific settings, all visible in the launch command:

Clip-Higher: asymmetric Proximal Policy Optimization (PPO) clipping (clip_ratio_low=0.2, clip_ratio_high=0.28), so the upper clip is loosened to preserve exploration on rare, high-reward tokens.

KL-free objective: use_kl_loss=False, kl_coef=0.0. DAPO drops the KL penalty entirely and lets the clip bounds do the regularizing.

Token-level loss aggregation: loss_agg_mode=token-mean, which weights long and short responses by their token counts rather than per sequence.

Overlong reward shaping: the dapo reward manager with an overlong buffer (overlong_buffer.len=4096, penalty_factor=1.0, max_resp_len=8192) that softly penalizes responses approaching the length cap instead of hard-truncating their reward.

Large GRPO groups: n_resp_per_prompt=16 samples per prompt, train_prompt_mini_bsz=32.

The async knobs mirror the geo3k example (staleness_threshold=0.1, trigger_parameter_sync_step=4, partial_rollout=True) but with require_batches=4, and the run is bounded by total_rollout_steps=$((512*100)).

In contrast to the geo3k run, this DAPO configuration—clip-higher, a KL-free objective, and overlong reward shaping—produces real learning: AIME-2024 accuracy climbs while response length stays stable.

The following table summarizes the hardware and software configuration used for this run.

rocm/verl:verl-0.7.1.amd0_rocm7.0.2_ubuntu22.04_py3.12_vllm0.20.2

Qwen2.5-Math-7B (approx. 15 GB, 4 safetensors shards)

BytedTsinghua-SIA/DAPO-Math-17k (train, approx. 299 MB parquet) + BytedTsinghua-SIA/AIME-2024 (val, approx. 29 KB)

acc@1 0.0 → 0.262 peak, 0.242 final

Steps 1 (pull the image) and 2 (launch the container) are identical to the GRPO walkthrough above, so this section picks up at Step 3.

The prepare script downloads the two parquet datasets and the model, then patches config.json. It is a required step that the script header and the release notes both call out:

The script lives at verl/experimental/fully_async_policy/shell/dapo_7b_math_fsdp2_4_4.sh. The resource split and async settings are the same idea as geo3k, but the trainer is FSDP2 rather than Megatron:

val_before_train=True runs one validation pass before any optimization (step 0), giving a clean baseline.

The run completes after 100 trainer steps (400 global steps) in 35,901.66 s (approx. 10 h), terminating on the total_rollout_steps budget:

GRPO vs. DAPO at a Glance#

Here is a summary of the experiments derived from the logs collected through internal experimentation as well as rocm-smi outputs.

Idle ratio is the fraction of trainer wall-clock spent waiting on the sample queue, parsed from the FullyAsyncTrainer step logs, while peak throughput is the maximum per-step tokens/GPU/s observed over the run, computed over the 4 training GPUs.

GRPO (Qwen2.5-VL-7B)

DAPO (Qwen2.5-Math-7B)

approx. 11 h (39,739 s)

approx. 10 h (35,902 s)

approx. 1,050 tokens/GPU/s

approx. 2,994 tokens/GPU/s

0.0 (plumbing run)

0.0 → 0.262 peak (0.242 final)

grew into truncation (clip approx. 0.77)

stable approx. 800–890 (clip approx. 0.0)

The throughput gap is mostly attributable to differences in modality and backend: the text-only Math-7B with an FSDP2 trainer and shorter, well-bounded responses keeps the rollout and update pipeline fuller than the heavier multimodal Megatron path.

Performance Comparison of Synchronous Use Case: AMD Instinct MI355X vs. NVIDIA B300#

For the throughput analysis, this section shifts from the async walkthroughs to a separate, synchronous benchmark. These runs use Qwen2-7B in a conventional colocated synchronous setup to derive throughput. The first experiment uses Proximal Policy Optimization (PPO) on Qwen2-7B, while the second uses Group Relative Policy Optimization (GRPO) on the same Qwen2-7B model. More details on the experiments and how to run them can be found at Reinforcement Learning from Human Feedback on AMD GPUs with verl and ROCm 7.0.0.

Both experiments are run with Tensor Parallelism (TP) = 2. In the GRPO configuration, n denotes the number of responses sampled per prompt (the group size); GRPO requires n > 1 because its advantage estimate is computed across the group. This is a key difference from PPO: PPO trains a separate critic model to estimate the value (expected return) used as the advantage baseline, whereas GRPO removes the critic and instead derives the baseline directly from the group: normalizing each response’s reward by the group’s mean (and standard deviation) to compute its advantage.

The results of this experiment are summarized in the table and visualized in the figure below.

Figure 1: Throughput (tokens/GPU/s) comparison of AMD Instinct MI355X vs. NVIDIA B300 on PPO and GRPO Qwen2-7B workloads at baseline.

In these benchmarks, on both Qwen workloads, MI355X measured higher throughput than B300 at default settings.

The async split sustains high GPU utilization. Dedicating 4 GPUs to vLLM rollout and 4 to the trainer kept the training GPUs busy roughly 91–94% of the time across both runs (about 93–94% for GRPO and 91–92% for DAPO), consistent with the log-derived idle ratios above, with healthy queues and zero dropped stale samples throughout.

DAPO on Math-7B learns. With clip-higher, a KL-free objective, and overlong reward shaping, AIME-2024 accuracy climbed from 0.0 to a peak of 0.262 (0.242 final) in 100 steps while response length stayed stable. This is a contrast to the geo3k plumbing run.

The same async knobs port across backends. staleness_threshold, trigger_parameter_sync_step, and partial_rollout behaved identically on Megatron and FSDP2. Together, they govern how off-policy your training is and how much rollout work you preserve across weight syncs, making the async approach more flexible than one-step off-policy training.

Match the reward to the response budget. DAPO’s buffer kept responses off the truncation wall and let accuracy climb. The geo3k run, without length-aware shaping, was rewarded into truncation instead. Separate quality runs from plumbing runs, and add length-aware rewards before judging model quality.

Higher throughput on MI355X. In these synchronous benchmarks at default settings, MI355X showed higher throughput on both Qwen workloads. See footnotes [1] and [2] for methodology.

This walkthrough demonstrates verl’s Fully Async Policy trainer end-to-end on AMD Instinct™ MI355X GPUs with ROCm. In this setup, physically separating the rollout and training fleets lets generation and weight updates run in parallel so the training GPUs are not stalled waiting on long-tail rollouts and sustain high GPU utilization. By dedicating 4 GPUs to vLLM rollout and 4 to the trainer, the training GPUs were busy approx. 91–94% of the time across both runs.

The walkthrough shows two examples that share the same async machinery but differ in model, data, algorithm, and trainer backend: GRPO on Qwen2.5-VL-7B-Instruct (Geometry3k vision-math, Megatron trainer) and DAPO on Qwen2.5-Math-7B (DAPO-Math-17k / AIME-2024, FSDP2 trainer). The same async knobs (staleness_threshold, trigger_parameter_sync_step, and partial_rollout) port cleanly across both Megatron and FSDP2, and full CPU offload keeps a 7B model comfortable on just 4 training GPUs.

The authors would also like to acknowledge the broader AMD team whose contributions were instrumental in this work: Jared Bowden, Dong Li, Gowtham Ramesh, Jiang Liu, Zhenyu Gu, Zicheng Liu, Emad Barsoum, Marco Grond, Lillian Zheng, Ramesh Mantha, Wenbo Shao, Pei Zhang, Matthew Steggink, Gazi Rashid, Bhavesh Lad, Pankaj Gupta, Aakash Sudhanwa, Joseph Macaranas, Kiran Thumma, Ian Dass, Ram Seenivasan, Amit Kumar, Anisha Sankar, Saad Rahim, Ehud Sharlin, Liam Berry, Cindy Lee, Lindsey Brown, Catherine Ortega, Ashley Cowart, Keith Anderson, Lorelei Misajlovich, Jennifer Barry.

Third-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.

The information contained herein is for informational purposes only and is subject to change without notice. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD products are as set forth in a signed agreement between the parties or in AMD’s Standard Terms and Conditions of Sale. GD-18u.

©2026 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective owners. Certain AMD technologies may require third-party enablement or activation. Supported features may vary by operating system. Please confirm with the system manufacturer for specific features. No technology or product can be completely secure.

Text integral sub licența deschisă a publicației · 3653 cuvinte

analiza aiqbench

Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark

Subiecte conexe