ROCm 10.0.0 Support for Ray 2.58.0: Enabling Scalable AI Workflows

ROCm 10.0.0 Support for Ray 2.58.0: Enabling Scalable AI Workflows#

7 min read. | 1686 total words.

Distributed AI teams usually run out of GPUs before they run out of ideas. Training and inference capacity is the scarce, expensive resource, and much of the tooling wrapped around it assumes one vendor stack, so putting a different accelerator to work means rewriting whatever touches the hardware.

Ray is the layer worth building on precisely because it already hides that detail: placement, scaling, and fault tolerance live behind framework APIs rather than in device code. That has been true on AMD Instinct™ GPUs for several years. ROCm™ 10.0.0 with Ray 2.58.0 continues the same contract: existing Ray code runs on Instinct unchanged, from data loading through distributed training to large language model (LLM) serving, with scaling decisions staying in Ray APIs and placement groups rather than in vendor-specific glue. This release adds a validated stack around that contract: PyTorch 2.12, vLLM on ROCm, and current Instinct GPU models (SKUs), including MI355X.

The payoff is concrete, and it is the second half of this post: the distributed serving example brings up an OpenAI-compatible LLM endpoint on Instinct GPUs, with vLLM sharding the model across devices by tensor parallelism, in about one screen of ordinary Ray Serve code. Exactly one line in it names AMD hardware.

Ray is an open-source framework that simplifies the development of distributed applications. It is widely used for scaling Python workloads, including machine learning (ML) training, hyperparameter tuning, and reinforcement learning, across clusters of CPUs and GPUs.

What ROCm 10.0.0 + Ray 2.58.0 Means for Ray Users#

ROCm is AMD’s open software platform for GPU-accelerated computing. ROCm 10.0.0 and Ray 2.58.0 continue to support the same programming model on Instinct, requiring no vendor-specific glue. What this release adds is a newly validated combination of APIs with the current ROCm, PyTorch, vLLM, and Instinct SKUs.

Ray discovers Instinct GPUs automatically, and torch.cuda maps to ROCm and the Heterogeneous-compute Interface for Portability (HIP), so @ray.remote(num_gpus=1), ScalingConfig, placement groups, and LLMConfig behave the way they do on any other accelerator. There is no ROCm code path to maintain alongside an existing one, and no porting step for the Python that defines a pipeline. Migrating a workload is a scheduling and configuration exercise, not a rewrite.

Ray does not treat Instinct hardware as a generic GPU. MI300X, MI325X, and MI355X are native accelerator types, so a deployment can request a specific SKU the same way it would request any native accelerator types. That makes heterogeneous fleets tractable: capacity from several Instinct generations can sit in one cluster while individual workloads still land on the SKU they were tuned for. See Pinning workloads to an Instinct SKU for when to use that constraint and when to leave it off.

Compatibility claims are only useful if they extend to the whole path a real workload takes. The evidence in this post is the distributed serving example: Ray 2.58.0 on ROCm 10.0.0, with vLLM on ROCm, serving a tensor-parallel LLM behind an OpenAI-compatible endpoint. The ROCm Communication Collectives Library (RCCL) carries the tensor-parallel traffic between vLLM workers. Kernel and driver optimizations in this release reduce per-operation overhead and speed up host-to-device transfers, which is where continuously streaming data pipelines spend their time.

For reinforcement learning from human feedback (RLHF) training that uses Ray on ROCm 10.0.0 with verl, see verl on ROCm 10.0.0.

Reinforcement learning — see the Verl 0.9.0 on ROCm 10.0: Next-Generation RL Post-Training on AMD Instinct GPUs post for RLHF with Ray on ROCm 10.0.0

Reinforcement learning — see the Verl 0.9.0 on ROCm 10.0: Next-Generation RL Post-Training on AMD Instinct GPUs post for RLHF with Ray on ROCm 10.0.0

Refer to the upstream Ray use cases documentation for detailed tutorials using AMD GPUs and ROCm.

Install Ray 2.58.0 on ROCm 10.0.0 first, then confirm that Instinct GPUs are visible to Ray.

Ray can be installed on a single node or across a cluster. The steps below are the same either way: meet the host requirements, follow the ROCm Ray install guide, then run the cluster confirmation snippet.

Install Ray 2.58.0 from the official guide, which covers building a Docker image, installing on bare metal or a custom container, and building from source.

Ray 2.58.0 release branch

Ray 2.58.0 release branch

After installation, Ray discovers AMD Instinct GPUs automatically, and torch.cuda maps to ROCm and HIP, so the standard Ray APIs work unchanged. Confirm that the cluster is ready for general Ray usage: resources are visible, and a GPU task runs on an Instinct device.

If Ray reports 0 GPUs, stop here and fix device access (/dev/kfd, /dev/dri, and the video and render groups) before running GPU workloads.

Notice the accelerator_type:AMD-Instinct-MI355X-OAM entry alongside the GPU count in that output. That key is how Instinct hardware enters Ray’s scheduler, and workloads can require it. Ray 2.58.0 recognizes the current Instinct parts natively:

Ray Core exposes this as @ray.remote(accelerator_type=...); Ray Serve LLM exposes it as the accelerator_type field on LLMConfig. Either way it is a placement constraint, not a ROCm enable flag. ROCm works whether or not the constraint is set.

When to pin: Pin the SKU when a cluster contains several Instinct generations, when the cluster also has NVIDIA GPUs, or when a workload is tuned for the memory capacity of a specific Instinct SKU. Pinning also lets Ray infer the SKU-specific accelerator_config (memory class and placement-group hints) that it derives from the accelerator type.

When not to pin: On a homogeneous single-SKU node, leaving accelerator_type=None is correct and simpler. Replicas then request generic GPU capacity plus the tensor-parallel degree, which is all the scheduler needs when every device is identical. Pinning a SKU in that case only creates a value to update when the workload moves to different hardware.

The failure modes are worth knowing before choosing. A wrong or unrecognized string fails closed: Ray cannot place the deployment, and says so. None fails open: the workload runs somewhere, which on a mixed cluster may be the wrong somewhere. Neither is universally safer, so pick based on how uniform the cluster is.

This is the payoff: a production-shaped, OpenAI-compatible LLM endpoint running on Instinct GPUs, in roughly one screen of code. Ray Serve LLM (ray.serve.llm) handles the HTTP surface and replica placement, vLLM is the engine, and the model is sharded across devices by tensor parallelism with RCCL (exposed through the NCCL backend) carrying the collectives.

Everything below is ordinary Ray Serve LLM code. The same script would run on any accelerator Ray supports. The one line specific to AMD hardware is the accelerator_type string, and even that is optional on a homogeneous node.

The example exercises the whole path end to end: Ray discovers Instinct GPUs, Serve places a replica, vLLM shards the model across GPUs, and a real chat-completions request returns a non-empty response, so a successful run means the serving stack works, not just that the import succeeded.

A large /dev/shm (around 10 GB or more). The default 64 MB shared-memory size is too small for RCCL among tensor-parallel workers

A large /dev/shm (around 10 GB or more). The default 64 MB shared-memory size is too small for RCCL among tensor-parallel workers

In the same verified environment, create ray_serve_vllm.py with the following contents:

The default configuration serves Qwen/Qwen2.5-0.5B-Instruct with tensor_parallel_size=2. Override the model, GPU shard count, HTTP port, or startup timeout with environment variables, where hf_model_id is the Hugging Face model ID, tensor_parallel_size is the number of GPUs to shard across, ready_timeout_s is the endpoint wait time in seconds, and serve_port is the Serve HTTP port:

The attention and key-value (KV) head counts of the model must be divisible by SERVE_TP_SIZE. Increase SERVE_READY_TIMEOUT_S on the first run so the replica can finish downloading weights and become ready.

accelerator_type="AMD-Instinct-MI355X-OAM" pins the replica to MI355X. Swap in AMD-Instinct-MI300X-OAM or AMD-Instinct-MI325X-OAM for those parts, or drop it to None on a homogeneous node. See Pinning workloads to an Instinct SKU.

accelerator_type="AMD-Instinct-MI355X-OAM" pins the replica to MI355X. Swap in AMD-Instinct-MI300X-OAM or AMD-Instinct-MI325X-OAM for those parts, or drop it to None on a homogeneous node. See Pinning workloads to an Instinct SKU.

A fail-fast check against ray.cluster_resources()["GPU"] so Serve does not sit UNHEALTHY waiting for GPUs that are not on the node.

A fail-fast check against ray.cluster_resources()["GPU"] so Serve does not sit UNHEALTHY waiting for GPUs that are not on the node.

enforce_eager=True and a modest max_model_len keep the example quick to start; both are ordinary vLLM engine arguments to tune for a real deployment.

enforce_eager=True and a modest max_model_len keep the example quick to start; both are ordinary vLLM engine arguments to tune for a real deployment.

If Ray reports fewer GPUs than SERVE_TP_SIZE, confirm ROCm device access (/dev/kfd and /dev/dri) and that the process is in the video and render groups. If the HTTP client times out, wait for the first Hugging Face download to finish or raise SERVE_READY_TIMEOUT_S.

In this blog you explored what ROCm 10.0.0 and Ray 2.58.0 deliver for Ray workloads on Instinct. You saw existing Ray code run unchanged: Ray discovers Instinct GPUs automatically, torch.cuda maps to ROCm and HIP, and placement stays in @ray.remote, ScalingConfig, placement groups, and LLMConfig. You walked through a tensor-parallel vLLM deployment behind an OpenAI-compatible endpoint, written as ordinary Ray Serve LLM code, on a stack validated with PyTorch 2.12 and vLLM on ROCm. One line in that example names the hardware, and on a uniform node you can leave it unset.

AMD remains committed to ongoing zero-day support for ROCm and Ray releases, so Instinct stays aligned with each new version as it ships. Keep checking the Install Ray on ROCm guide for the latest supported combination. The guide is updated as new ROCm and Ray versions are validated, and it is the reference to confirm before you install or upgrade.

Our team plans to keep publishing walkthroughs as that support moves forward, covering more of the Ray workloads this stack already runs, including distributed training and batch inference. For reinforcement learning from human feedback on ROCm 10.0.0, see the verl on ROCm 10.0.0 post. Follow the ROCm blogs for the next installments. Install Ray 2.58.0 from the guide above, confirm the cluster, and run the serving example so an existing Ray workload can land on Instinct today.

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, AMD Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved