AMD GPU Operator v1.5.0: DRA Support, Automated GPU Node Recovery, and Expanded Kubernetes Infrastructure Control

GPU Operator v1.5.0 introduces several major infrastructure capabilities for Kubernetes-based AMD GPU deployments, including support for Kubernetes Dynamic Resource Allocation (DRA), automated GPU node remediation workflows, and Node Problem Detector (NPD) integration.

AMD GPU Operator v1.5.0: DRA Support, Automated GPU Node Recovery, and Expanded Kubernetes Infrastructure Control#

6 min read. | 1581 total words.

In addition, the managed stack is updated to ROCm 7.2.1 and includes multiple Device Metrics Exporter enhancements, including new ECC metrics, per-process Compute Unit occupancy telemetry, configurable health polling, and lower-latency Unix socket communication between GPU Agent and the exporter.

In this blog, you’ll get a practical look at what’s new in AMD GPU Operator v1.5.0 and why this release matters for teams running AMD GPUs on Kubernetes. You’ll walk through how Dynamic Resource Allocation (DRA) changes the way GPUs are scheduled through DeviceClass and ResourceSlice, how Auto Node Remediation brings Device Metrics Exporter, Node Problem Detector, and Argo Workflows together into a real recovery loop, and how the new KMM, kubelet socket, and image pull secret controls make the Operator far easier to fit into existing cluster setups. You’ll also see the Device Metrics Exporter improvements that give you sharper visibility into GPU behavior, including per-process CU occupancy, deferred ECC counters, and a leaner Unix socket path between the GPU Agent and exporter.

Prior to v1.5.0, GPU Operator exposed AMD GPUs exclusively through the Kubernetes Device Plugin framework. GPUs were advertised through the kubelet device plugin API and exposed as allocatable resources directly to the scheduler.

GPU Operator v1.5.0 introduces support for Kubernetes Dynamic Resource Allocation (DRA) through the AMD GPU DRA Driver.

DRA changes how accelerators are represented and allocated inside Kubernetes. Instead of statically advertising devices directly through kubelet, DRA introduces Kubernetes-native resource allocation objects such as DeviceClass and ResourceSlice, enabling scheduler-driven allocation workflows and more flexible resource negotiation.

Unlike the traditional Device Plugin model, DRA enables multiple containers within a pod to coordinate access to the same GPU resources. This is particularly useful for tightly coupled AI workflows where inference containers, preprocessing pipelines, or sidecar services need shared accelerator access.

Enable DRA through GPU Operator: set spec.draDriver.enable: true in the DeviceConfig CR and let GPU Operator deploy and manage the DRA driver components.

Install the DRA driver standalone: follow the Helm installation commands in the AMD GPU DRA Driver v1.0.0 release notes.

resource.k8s.io/v1

resource.k8s.io/v1beta2

resource.k8s.io/v1beta1

Kubernetes 1.32–1.35

without requiring manual API version selection.

This validation is important because both systems expose GPU resources differently. Running both simultaneously would create conflicting allocation paths and duplicate device advertisement inside Kubernetes.

For OpenShift environments, GPU Operator automatically creates the required RBAC resources and SecurityContextConstraints (SCCs) for the DRA Driver deployment.

Kubernetes 1.32+

Operationally, DRA support aligns AMD GPU infrastructure with Kubernetes’ newer resource allocation model instead of relying exclusively on static kubelet device advertisement.

GPU Operator v1.5.0 introduces Auto Node Remediation (ANR), enabling automatic recovery workflows for unhealthy GPU worker nodes.

In distributed AI environments, GPU node instability is particularly disruptive because training workloads often rely on tightly synchronized collective communication patterns. A single unhealthy worker can interrupt all-reduce operations, degrade training throughput, or force entire workloads to restart.

Prior to v1.5.0, remediation typically required manual operational steps:

This integration allows GPU telemetry to directly influence Kubernetes node lifecycle management instead of remaining isolated inside monitoring dashboards or Prometheus metrics.

Limits the number of simultaneous remediation events occurring in the cluster, preventing cascading node recovery operations during broader infrastructure instability.

Controls workload eviction behavior prior to remediation.

Prevent multiple nodes from entering remediation simultaneously within defined intervals.

Allow unhealthy nodes to be isolated from scheduler placement during recovery workflows.

and requires Argo Workflows v4.0.3.

This simplifies integration into enterprise Kubernetes environments with stricter security controls.

GPU Operator v1.5.0 significantly expands deployment flexibility for heterogeneous Kubernetes environments.

Prior to v1.5.0, setting:

This made it difficult to integrate GPU Operator into environments where KMM was already deployed and managed externally.

v1.5.0 introduces independent controls:

GPU Operator v1.5.0 also introduces support for custom package repository URLs and custom GPG key URLs during in-cluster driver image builds:

reducing duplicated registry configuration in private-registry environments.

GPU Operator v1.5.0 also includes several significant Device Metrics Exporter (DME) enhancements.

instead of TCP/IP.

while improving communication efficiency between the GPU Agent and exporter.

Per-process Compute Unit occupancy telemetry with process_id labeling.

for multi-process AI and HPC workloads.

Deferred ECC counters for ECC-supported blocks.

before they escalate into fatal GPU failures.

KFD_PROCESS_ID labeling is now disabled by default to reduce Prometheus cardinality overhead.

because each GPU/process combination generates unique metric series across every scrape interval.

Operators requiring process-level attribution can still re-enable the label through the exporter ConfigMap.

Consider a multi-node MI300 training cluster running Kubernetes 1.34 with DRA enabled.

This creates a tighter operational loop among telemetry, scheduling, remediation, and workload recovery than was possible in previous GPU Operator releases.

than previous GPU Operator releases.

GPU Operator v1.5.0 significantly expands the operational scope of AMD GPU infrastructure in Kubernetes.

DRA support introduces a newer Kubernetes-native GPU allocation model built around DeviceClass and scheduler-driven resource allocation. ANR and NPD integration move GPU failure handling into automated remediation workflows tied directly to Kubernetes node health state.

Combined with enhanced telemetry from Device Metrics Exporter and updated ROCm 7.2.1 stack support, v1.5.0 strengthens how platform teams deploy, monitor, and recover GPU infrastructure in production AI and HPC clusters.

GPU Operator Documentation: https://instinct.docs.amd.com/projects/gpu-operator/en/latest/

AMD GPU DRA Driver v1.0.0 Release: ROCm/k8s-gpu-dra-driver

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information.

However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, [insert all other AMD trademarks used in the material here per AMD Trademarks] and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. [Insert any third party trademark attribution here per AMD’s Third Party Trademark List.] © Advanced Micro Devices, Inc. All rights reserved

aiqbench analysis

AMD GPU Operator v1.5.0: DRA Support, Automated GPU Node Recovery, and Expanded Kubernetes Infrastructure Control

Related coverage