Topology-Aware Workload Scheduling with NVIDIA Topograph

AI factories are power-limited systems that deliver maximum value when fully optimized. GPU workload placement is a key optimization. Poor workload placement fragments topology domains and forces traffic across shared links, reducing throughput, raising job costs, and leaving GPUs consuming provisioned power while waiting on data without advancing the workload.

GPUs exchange data continuously during training and inference, so distributed workloads benefit from communication locality. NVIDIA NVLink and NVLink Switch provide high-bandwidth, all-to-all scale-up connectivity within rack-scale GPU domains, while NVIDIA Spectrum-X Ethernet provides predictable, low-latency scale-out networking across systems and racks.

A scheduler can place workloads efficiently only with a current, accurate view of those GPU and fabric relationships, and keeping that view current as the cluster changes is where placement breaks down in practice.

NVIDIA Topograph solves that problem. It discovers cluster topology from cloud APIs or on-premises fabric systems, normalizes it into a common model, and publishes it in the format each workload manager expects: Kubernetes node labels, Slurm topology configuration, or Slinky ConfigMaps. Within the NVIDIA DSX OS cluster orchestration layer, Topograph works alongside Dynamic Resource Allocation (DRA) and KAI Scheduler to enable topology-aware gang scheduling across AI factory infrastructure.

This post walks through deploying Topograph and using it to schedule topology-aware workloads on Kubernetes, Slurm, and Slinky.

Topograph maps how cluster hardware is connected so schedulers can favor nearby resources. Think of the network as a road system: GPUs within the same locality domain have short, high-bandwidth paths, while traffic between domains crosses more shared links and switches. Spreading a tightly coupled workload across distant domains can increase contention and latency, so Topograph helps place workloads in the most efficient locations and avoid these bottlenecks.

Modern NVIDIA Quantum InfiniBand ports can achieve up to 800 Gb/s, while NVIDIA NVLink provides 1.8 TB/s of bidirectional bandwidth per GPU in its fifth generation (NVIDIA Blackwell, such as GB200/GB300) and 3.6 TB/s per GPU in its sixth generation (Vera Rubin), through a dedicated NVIDIA NVLink Switch fabric. That non-blocking, all-to-all design gives each GPU its own lane rather than sharing bandwidth under load. Schedulers with a current view can favor GPUs in the closest topology domain.

Slurm and Kubernetes both support topology-aware allocation, but a scheduler can only act on the topology it observes. Topograph regenerates that view on request and upon watched cluster changes, so the scheduler works from current data rather than a manually maintained snapshot.

Topograph is an open source toolkit that identifies a cluster’s network topology, enabling workload managers to make topology-aware scheduling decisions. It has two concepts: providers and engines. A provider discovers topology from cloud APIs or on-premises systems and normalizes it into a canonical model. An engine translates that model into Slurm configuration, Kubernetes labels, Slinky ConfigMaps, Node Feature Discovery (NFD) resources, or instance-oriented topology JSON.

Cloud providers that have a working integration with Topograph include Google Cloud, Lambda, Nebius, Nscale, and OCI, with more cloud and colocation providers in development.

Table 1. Supported topology providers by engine

Scope and interpretation. This matrix reflects current upstream main as of September 16, 2026. It shows supported provider-to-engine output combinations; requirements can vary by Topograph version, environment, and provider configuration.

The Crusoe provider reads fabric and accelerator-domain labels from Crusoe Managed Kubernetes nodes; Topograph therefore runs in Kubernetes for this provider.

The Slurm engine can run in Kubernetes, but it requires a writable volume for its configured topology.conf output path.

The NFD engine requires the alpha NodeFeatureGroupAPI feature gate. The Kubernetes engine publishes Node labels instead.

Figure 1. Topograph accepts a generation request, discovers and normalizes topology from a selected cloud or network fabric provider, and publishes scheduler-ready outputs for Slurm and Kubernetes or instance-oriented topology JSON. Kubernetes deployments can also use runtime helpers to react to cluster changes and collect per-node data

POST /v1/generate – submits an asynchronous request and returns its ID with HTTP 202.

GET /v1/topology?uid=<request-id> – returns HTTP 202 while processing and HTTP 200 with the result when complete.

POST /v1/lookup – returns the cached status or result for the same request body without submitting it again.

GET /healthz – is the liveness endpoint.

GET /metrics – exposes Prometheus metrics.

The aggregation delay is required; 15 seconds is typical. Repeated identical requests reset a trailing timer and are processed once, reducing redundant work during bursts of cluster events.

For testing without production hardware, simulation models describe node and switch hierarchies. The kwok-nodes utility and Kind/KWOK helpers turn those models into virtual Kubernetes nodes.

The default Kubernetes scheduler doesn’t discover physical interconnect hierarchy. Topograph addresses that gap by publishing provider-reported topology as node labels, which native affinity and topology-aware schedulers can consume.

Prerequisites are Kubernetes 1.27 or later, Helm 3.10+ or 4.x, kubectl permissions, and a supported provider. KAI Scheduler or Kueue TAS is optional for topology-aware gang scheduling.

Replace <provider> with the value that matches your environment.

The repository includes example Helm values files in charts/topograph, named with a values.k8s prefix and a short scenario description. Each carries inline configuration comments.

The bundled test hooks query /healthz and /metrics in-cluster and confirm the responses include the topograph_version metric.

Fabric tier 0 is the leaf switch closest to the compute node, and tier numbers increase outward. Topograph writes only the tiers present in the discovered topology, with no fixed maximum depth. Operators can set the Kubernetes engine’s fabricLabels array and acceleratorLabel parameter to use custom keys; tiers beyond that array are not labeled. The sub-domain key is fixed.

NOTE: Topograph reflects reported rather than intended topology. Labels refresh when generation runs, for example, after a watched node or pod change. Visibility of a fabric change depends on the provider and its triggering events.

The API is a ClusterIP service by default. With the release and namespace above, its address is: topograph.topograph.svc.cluster.local:49021.

Each matching term contributes to a candidate node’s score, strongly favoring the tier-0 domain of existing app=myapp Pods while also rewarding tier-1 locality. Because the default scheduler places Pods individually, this is a preference rather than globally optimal gang placement.

KAI Scheduler and Kueue can use the same node labels for topology-aware gang placement. Kubernetes 1.36 also introduced alpha topology-aware workload scheduling through KEP-5732. Upstream beta work is ongoing; consult the enhancement tracker rather than depending on a specific future release.

Apply it with kubectl apply -f cluster-topology.yaml, then annotate a multi-Pod Job:

The required annotation keeps the gang within a single tier-1 domain. The preferred annotation asks KAI to concentrate Pods in a tier-0 domain when feasible, but permits multiple tier-0 domains inside the required boundary.

For more advanced topology-aware scheduling examples, see the documentation for Grove and NVIDIA Dynamo.

Grove provides Kubernetes APIs and an operator for hierarchical gang scheduling, topology-aware placement, and coordinated scaling. Dynamo is an open source distributed inference serving framework that integrates with Grove for Kubernetes workload orchestration.

Topograph also supports consumers already using Node Feature Discovery. The nfd engine publishes one NodeFeature per selected topology node and one NodeFeatureGroup for every distinct fabric-tier, XCLR-domain, and XCLR-sub-domain value. The NFD master evaluates those specifications and owns each group’s status.nodes membership.

Install nfd first with its alpha NodeFeatureGroupAPI feature gate enabled; it is off by default. Then select the engine and the namespace where the NFD master runs:

Use this output when a downstream component consumes NodeFeatureGroup objects; it is not a substitute for Kubernetes topologyKey labels. For native Pod affinity, KAI Scheduler, or Kueue TAS, continue to use engine: k8s. The chart scopes NFD permissions to the nfd namespace. The engine deletes stale Topograph-managed objects after reconciliation, but preserves the last published topology if a generation produces none.

Topograph generates cluster-wide configurations in the tree and block formats, shown in the top-center and bottom-center panels of Figure 2, below. Slurm 25.05 introduced per-partition configuration in YAML format, which Topograph also supports, as shown in the diagram.

Figure 2. A representative three-tier cluster topology with Multi-Node NVLink domains and Topograph’s configuration-dependent Slurm output modes: cluster-wide tree, cluster-wide block, or per-partition topology YAML

Slurm clusters typically run on Linux bare-metal servers or virtual machines, where Topograph is installed via a native package manager. The repository includes Debian and RPM build targets:

The package installs the service without starting it, so you can review and edit the configuration file /etc/topograph/topograph-config.yaml

Replace <provider> with the value that matches your environment.

To initiate discovery, POST to Topograph’s /v1/generate endpoint, which regenerates the Slurm topology configuration.

Use topology/block plus optional blockSizes for block output.

The optional reconfigure parameter runs scontrol reconfigure after a file is written and defaults to false. If topologyConfigPath is omitted, Topograph returns the generated content from the result endpoint instead of writing a file.

It registers a permanent strigger for node up and down transitions. It does not detect arbitrary switch rewiring or every inventory change.

Slinky, developed by SchedMD, runs Slurm on Kubernetes. NVIDIA acquired SchedMD in December 2025. The Topograph Slinky engine maps Kubernetes nodes to slurmd Pods and writes Slurm topology data to a ConfigMap.

The Slinky engine supports cluster-wide topology/tree and topology/block output, as well as multiple-topology YAML for partition-specific configurations.

The repository provides ready-to-adapt Helm examples for tree, block, per-partition, and InfiniBand block deployments.

Topograph regenerates and updates the ConfigMap when selected slurmd Pods change.

The dra provider is a narrower Slinky block-topology option for MNNVL systems. It reads existing nvidia.com/gpu.clique labels when regenerating the topology configuration.

For dynamic Slurm nodes, the optional useDynamicNodes mode also annotates selected Kubernetes nodes with the current Slurm topology specification. ConfigMap updates and dynamic-node reconciliation are distinct mechanisms, so choose the mode that matches the deployed Slinky configuration.

Placement problems compound at scale and surface as network congestion. Topograph gives schedulers a current, provider-reported map of the physical network, so topology-aware decisions stay consistent across cloud and on-premises environments without manual maintenance.

Through KAI Scheduler, Kueue, and native Kubernetes, the map improves AI factory efficiency, tokens per watt, and cost.

Deploy Topograph from the dsx-ai-factory/topograph GitHub repo, or learn more about the DSX OS ecosystem.