Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to discover the problem only after the workload is running.

These issues can appear at any point in a cluster’s lifecycle. A GPU cluster that passes readiness checks at handover changes as nodes are replaced, drivers are patched, and capacity is added. Without repeated validation, the original certification no longer reflects the cluster’s condition.

This visibility gap wastes compute on failing nodes and leaves healthy nodes idle out of caution. Utilization metrics can mask these losses. Goodput, which is the volume of work successfully completed, provides a more useful measure of cluster efficiency. Maintaining cluster readiness requires validation throughout the cluster’s operational lifecycle—after configuration changes, as capacity grows, and before repaired nodes return to service.

The NVIDIA DSX AI Factory Platform brings together technologies for designing, building, and operating efficient, resilient AI factories. Its open source software layer, NVIDIA DSX OS, includes tools for configuring, certifying, and maintaining the health of accelerated Kubernetes infrastructure.

Configure: NVIDIA AI Cluster Runtime (AICR) establishes a reproducible baseline for GPU-optimized Kubernetes infrastructure. Defines version-locked configuration recipes for components such as networking, storage, and drivers. Renders recipes as bundles for existing deployment tools.

Defines version-locked configuration recipes for components such as networking, storage, and drivers.

Renders recipes as bundles for existing deployment tools.

Certify: NVIDIA Cluster Readiness Engine (NVCRE) validates whether the cluster is ready to run production workloads. Tests GPUs, networking, and storage using AI workloads and diagnostics. Records certification results and helps identify faulty nodes.

Tests GPUs, networking, and storage using AI workloads and diagnostics.

Records certification results and helps identify faulty nodes.

Operate: NVSentinel monitors cluster health and supports fault remediation and revalidation. Publishes node conditions based on GPU, node, and network health. When configured, cordons or taints unhealthy nodes, drains workloads, and remediates faults. Can trigger revalidation with NVCRE before nodes return to service.

Publishes node conditions based on GPU, node, and network health.

When configured, cordons or taints unhealthy nodes, drains workloads, and remediates faults.

Can trigger revalidation with NVCRE before nodes return to service.

All three are open source projects licensed under Apache 2.0 and welcome external contributions.

Before you begin, install and configure AICR, NVCRE, and NVSentinel using their installation guides. The examples assume access to a Kubernetes GPU cluster with the required CLIs, controllers, dependencies, and NVCRE test provider configured.

This walkthrough follows one cluster through its lifecycle: configuration from a recipe, certification before production use, monitoring during operation, and recertification after repair. Each command is a step in that sequence.

AICR defines the cluster’s software baseline in version-controlled files and renders them into artifacts your existing deployment tooling applies. It generates and validates; your deployer applies and reconciles them. Declarative templates help detect configuration drift from manual changes and keep node configurations consistent after repairs.

You describe the target environment. AICR resolves it against a library of validated overlays and emits a recipe:

Pinned versions: The version is fixed, helping you reproduce the software configuration and investigate failures. In contrast, a floating tag can point to a different software version over time, making failures harder to reproduce.

A dependency graph: dependencyRefs declares what must be installed and healthy first. AICR resolves the graph when it renders the bundle, so the ordering is already encoded in the artifacts your deployer applies. You don’t need to specify that cert-manager comes before NVSentinel, and adding a component later doesn’t require renumbering.

Health assertions per component: Each entry carries a healthCheckAsserts block that runs after installation and checks whether the component meets its configured health criteria. Kubernetes can report a pod stuck in CrashLoopBackOff as Running, so a check based only on the pod phase can miss a broken component. The shipped assertions filter on the container waiting reason rather than the pod phase alone.

--deployer accepts helm, argocd, argocd-helm, flux, or helmfile, so the same validated recipe becomes Argo CD Application manifests, Flux resources, a Helmfile release graph, or plain Helm commands. You can keep your existing deployment tooling and use AICR to generate its configuration artifacts.

Because the recipe is a file, you can use version history, code review, and drift detection to manage the baseline.

--fail-on-drift turns the drift check into a continuous integration (CI) gate rather than a report.

The cluster is now built to a known baseline. It still needs workload validation. Standard health checks confirm that basic services are running. They don’t confirm that the cluster can run a demanding multinode job across its interconnect without stalling. NVCRE validates workload readiness by running training jobs and NVIDIA Collective Communication Library (NCCL) tests.

NVSentinel manages validation requests, scheduling gates, quarantine, and repair, while NVCRE runs tests and records results. Table 1 shows how these responsibilities fit into the node validation lifecycle.

1. Request and gate

Create a ValidationRequest for new or repaired nodes. After upgrades or fabric changes, operators cordon the nodes and create the request.

A cordon or taint blocks scheduling on the nodes during validation.

2. Test and attribute

Run workloads and diagnostic tests. Bisect failing groups to identify suspect nodes.

Test results identify suspect nodes. NVCRE doesn’t change the node state.

3. Record and evaluate

NVCRE records pass/fail results, failing node IDs, and failure reasons in the Certification resource. The lifecycle manager evaluates the results.

The validation outcome determines whether the scheduling gate can be removed.

4. Resolve the gate

Remove or retain the scheduling gate based on validation results and any remaining tests.

Pass: remove the configured gate once no other tests hold the node. Failure after retries: keep the node cordoned.

Table 1. Node validation duties, actions, and outcomes

The report shows each category’s status, runtime, measured bandwidth, and the reason for any failed nodes.

Thresholds make certification a performance gate. None are configured by default. Without the options.thresholds block above, a run reports its measurements and fails when the tests can’t start. A node performing below the target can pass. Thresholds help catch performance degradation rather than breakage.

The catalog covers NCCL communication variants, the NVIDIA Data Center GPU Manager (DCGM) level 4 diagnostic suite, and pretraining workloads using NVIDIA NeMo Framework with 8-billion- and 56-billion-parameter models. A WorkloadRun resource turns your own containerized test into a certification check. NVCRE infers the rest from the target nodes, so the manifest stays concise on any supported platform.

These tests can help identify performance issues that aren’t apparent in telemetry. A GPU running just below its performance target might not produce Xid errors, breach monitoring thresholds, or cause node metrics to appear abnormal. That one GPU can slow a synchronous training job to its slowest rank, and operators may find out hours later, or only after a user complains. An NCCL bandwidth test can uncover this performance degradation.

Isolating the cause matters as much. If a 64-node all-reduce yields poor bandwidth, every node is suspect, and bisecting by hand can be time-consuming. NVCRE splits failing groups and reruns the tests until it reaches a minimum group size, then reports the suspect nodes with a reason for each.

Run NVCRE before handing a cluster or new capacity to a tenant and when repaired hardware rejoins the pool.

The certified cluster is now running workloads and needs to stay healthy. AICR and NVCRE provide point-in-time validation. NVSentinel runs continuously alongside workloads.

NVSentinel watches signals the cluster already produces: DCGM health checks, Xid errors, network interface card (NIC) states, and cloud provider maintenance events. It doesn’t consume GPU time, leaving GPUs available for workloads.

NVSentinel installs in monitoring mode by default: it reports faults as node conditions and won’t cordon, evict, or remediate nodes. You can roll it out on a live cluster while you evaluate its reports. You can enable quarantine, draining, and remediation once you trust the reported signals.

Detection publishes a health event; a Fault Quarantine ruleset decides what happens to the node. Rulesets use Common Expression Language (CEL) expressions in your values file:

A cordon or taint provides a signal that your scheduler, alerting, and lifecycle tooling can use without integrating directly with NVSentinel.

A circuit breaker limits the impact by capping how many GPU nodes can be quarantined in a sliding window. The Fault Quarantine configuration guide covers the bounds and the rest of the ruleset options.

Not every fault takes the same path. A known fatal error goes straight to remediation. The health events analyzer handles other cases by detecting patterns across a node’s failure history, so a node that fails repeatedly becomes eligible for investigation based on the accumulated record rather than any single event.

A health event with the recommended action CONTACT_SUPPORT requires manual investigation or support intervention.

A node fails, gets repaired, and needs to rejoin the pool. Validation before the node returns to service closes the operational loop, but manual handoffs can interrupt the process. Manual certification produces a point-in-time result that routine operations can make outdated. Someone has to notice that the cluster changed, rerun validation, and then read the output and act on it. NVSentinel and NVCRE can help automate this process.

Two lifecycle events trigger revalidation automatically when configured. The NVSentinel lifecycle controller creates a ValidationRequest when a node joins the cluster and again when a node returns after remediation. It gates scheduling on the node using a cordon or taint, then uses NVCRE to run the tests and record the results. The controller evaluates those results and removes the gate after the tests pass, so a node that still fails validation cannot silently rejoin the pool. When you configure nvcre-provider as its test provider, those tests are NCCL or NVIDIA Nemotron runs through NVCRE:

The lifecycle-manager chart turns on new-node validation; the fault-quarantine flag opts in to post-remediation validation separately. schedulingGate.cordon sets the cordon as the scheduling gate for validation, and remove: true uncordons the node once its tests pass.

The process is different for in-place driver upgrades, operator upgrades, or fabric changes. These changes don’t automatically trigger validation. Because the baseline is a file, aicr diff --fail-on-drift fails a pipeline stage when a live cluster drifts from its recipe, providing a trigger for a certification run in the same pipeline that made the change.

The controller also acts on the result. If the request passes, it removes the scheduling gate once no other tests hold the node. If the test fails after its retries, the node remains cordoned.

A fault can also surface during validation: GPUs may start thermally throttling during a training test. NVSentinel quarantines the node, and the controller retries the test after remediation. The node stays gated until validation passes.

Recovery requires no manual node edits: a later request that passes the same tests clears the failure and removes the gate. Deleting all failed validation requests that reference the node can release it without rerunning validation.

The handoff has a clear boundary. NVCRE stops at attribution, recording which nodes failed and why; NVSentinel owns quarantine and repair. Two systems cordoning the same node could produce double-cordons and desynchronized cleanup.

This separation also enables you to use your own node lifecycle tooling. A platform team can consume the same certification results and take over NVSentinel’s role. Extending the stack means writing a ruleset or a policy, rather than forking a project.

AICR, NVCRE, and NVSentinel help operators make configuration, validation, and recovery repeatable. With remediation and lifecycle validation enabled, NVSentinel can coordinate repair and testing before nodes return to service, reducing manual handoffs. Table 2 outlines what these tools handle and what operators still manage.

AICR provides version-locked recipes and generates configuration artifacts for existing deployment tools. Operators can review deviations from a defined baseline.

Approve baseline changes, deploy the artifacts, run drift checks, and reconcile differences.

NVCRE runs selected workloads and diagnostics and records readiness results. With lifecycle validation configured, NVSentinel can coordinate testing before nodes return to service.

Select tests and thresholds appropriate to the cluster and investigate failed results.

When enabled, NVSentinel coordinates quarantine, draining, remediation, and revalidation, reducing manual handoffs between these actions.

Define quarantine rules, remediation limits, and escalation procedures; handle faults that automated remediation cannot resolve.

Operators can incorporate baseline checks and workload validation into their upgrade workflows.

Trigger validation after in-place driver, operator, or fabric changes; these changes do not automatically trigger revalidation.

Table 2. How AICR, NVCRE, and NVSentinel change operational tasks and which responsibilities remain with operators

Get hands-on with these tools at the NVIDIA DSX OS AI Inference and Contributor Workshop, held alongside KubeCon + CloudNativeCon North America. Separate registration is required, and seating is limited.

Visit NVIDIA booth 170 at KubeCon for more Kubernetes deep dives and live demos.

Join the DSX AI Factory Slack community to continue the conversation.

Explore open-source, modular components in the NVIDIA DSX AI Factory GitHub.