NVIDIA’s NodeWright is now open source after internal use

NVIDIA has released NodeWright, an open-source project designed to update entire fleets of GPU nodes without interrupting active training runs, a feat currently often managed with “a spreadsheet, a maintenance window, and an engineer watching a terminal at 3 a.m.” Unlike standard Kubernetes node management, NodeWright addresses the unique challenges of GPU infrastructure where hardware scarcity prevents simple node replacement and rescheduling of long-running jobs. The project, previously used internally at NVIDIA as Skyhook, is part of NVIDIA DSX OS, a portfolio of open-source tools including NVIDIA GPU Operator and Topograph, aiming to treat entire AI facilities as single, cohesive systems. NodeWright distinguishes itself from existing configuration tools by prioritizing fleet-level changes over individual machine management, a distinction important for large-scale AI infrastructure. Traditional systems like Ansible and Puppet were not designed to handle the complexities of actively running, sensitive workloads within a Kubernetes cluster; updating a kernel parameter across 200 GPU nodes requires more than simply executing a script. NVIDIA GPU Operator, NVIDIA Network Operator, Topograph, DRA Driver for NVIDIA GPUs, NodeWright, NVIDIA Cluster Readiness Engine (NVCRE), and NVSentinel all contribute to this AI-ready foundation,…