Achieving Peak System and Workload Efficiency on NVIDIA GB200 NVL72 with Slurm Block Scheduling

NVIDIA GB200 NVL72 introduces a fundamentally new way to build GPU clusters by extending NVIDIA NVLink coherence across an entire rack. This design enables exascale performance, but it also changes the assumptions that many scheduling systems were built on.

As a result, "rack-scale locality" becomes a hard constraint. When workloads cross domain boundaries, performance drops sharply, and a scheduler that treats the network fabric as a best-effort tree topology will fragment allocations in ways that increase queue times and degrade application performance.

To address this, Slurm workload manager introduced the topology/block plugin and continues expanding its capabilities with segmented scheduling. The plugin enables administrators and users to express application-specific NVLink requirements as atomic blocks rather than loosely optimized allocations.

This post explains how NVIDIA GB200 NVL72 architecture is unique, how Slurm block scheduling helps optimize placement and performance, and how to configure topology.yaml, --segment, and related features so you can move from prototype clusters to production-grade rack-scale orchestration.

How is NVIDIA GB200 NVL72 architecture unique?

NVIDIA GB200 NVL72 is an exascale computer in a single rack that represents a new paradigm in GPU cluster design. While previous generations of servers used NVIDIA NVLink within a single chassis, GB200 NVL72 extends this coherent memory domain across an entire rack: 72 NVIDIA Blackwell GPUs spanning 18 compute trays, unified with fifth-generation NVLink.

Figure 1. The increasingly complex interconnects of GB200 NVL72 rack and NVLink topology require more advanced workload intelligence

All communications within the rack operate at NVLink speeds: NVIDIA GB200 NVL72 provides 1.8 TB/s bidirectional throughput per GPU, for a total of 130 TB/s aggregate bandwidth. Communication crossing domain boundaries faces a steep performance drop, typically 50 GB/s (400 Gb/s) through InfiniBand or Ethernet.

Operating GB200 NVL72 clusters at scale requires new workload scheduling algorithms to treat NVLink domains as hard boundaries for jobs. While these new algorithms are essential for efficient workload, they also require administrative awareness around system fragmentation. The topology/block Slurm plugin helps users and administrators with both of these.

How does block scheduling work in Slurm?

Slurm has long supported topology-aware job scheduling: the topology/tree plugin has been the standard for large-scale clusters. It models the networking compute fabric as a hierarchical tree of switches and nodes. While the primary objective of topology/tree is to minimize the number of switches a job spans, it is a best-effort attempt. The job might end up being heavily fragmented across leaf switches in order to start sooner.

For clusters with an InfiniBand fabric connecting all compute nodes, this trade-off makes sense. A job split across multiple leaf switches might run slightly slower than under one single leaf switch, but the tradeoff between start time and performance is generally considered acceptable.

The introduction of GB200 NVL72 and GB300 NVL72 required a new approach. From a joint effort between NVIDIA and SchedMD, the new topology/block plugin was introduced in the Slurm 23.11 release to support rack-scale architectures such as GB200 NVL72.

Figure 2. Each multinode NVLink domain in the cluster is modeled as a block, which is a rigid scheduling unit

If a job submits an allocation request that fits within a single block (18 nodes or less), the nodes will always be allocated from one block and the job will not be fragmented.

Figure 3. Examples of job allocations (-N12, -N8, -N32) showing how nodes are assigned across blocks under default scheduling

With the default behavior of topology/block, a job requesting 16 nodes forces the scheduler to wait for a single multinode NVLink domain with 16 idle nodes. This can lead to increased job queueing times. In reality, the application might have NVLink connectivity requirements that are smaller than the full NVL72 domain.

To enable users to communicate the topology requirements of their applications to the job scheduler, Slurm introduced the --segment argument for topology/block jobs. This acts as a knob that defines the atomic group of nodes that must be placed in the same block and helps balance scheduler efficiency against the actual hardware locality requirements of the application.

By modifying this parameter, you can significantly ease scheduling constraints. For example, a job requesting 12 nodes but specifying --segment=4 can be split across three separate blocks. This new job would not be able to run without --segment=4, as there is no single block with 12 nodes available.

Figure 4. Segmented allocation example (-N12 --segment=4) showing how jobs can be split across multiple blocks

How does Slurm block scheduling optimize performance?

An important subtlety that often surprises users is the fact that Slurm can assign multiple segments of the same job to the same block.

Using segments is essential for optimizing performance based on the specific locality requirements of the workload: Tensor Parallelism (TP) may require small, tight segments to keep latency-sensitive communication on the high-speed NVLink fabric, while Expert Parallelism (EP) may require larger segment sizes to enforce that all-to-all collective operations will always be performed within a single NVLink domain.

Using a large segment value such as --segment=16 is also a way to yield a balanced allocation of the nodes across blocks. Figure 4 shows 32 nodes split as 18 nodes on block 3 and 14 nodes on block 4. Using --segment=16 instead would ensure exactly 16 nodes on each block, as shown in Figure 5.

Figure 5. Balanced allocation example (-N32 --segment=16) showing even distribution of nodes across blocks

The command-line arguments --consolidate-segments and --spread-segments introduced in Slurm 25.11 enable users to influence the placement of segments.

For topology/block, our recommendation for Slurm administrators is to define one block for each GB200 NVL72 domain (18 nodes) in the cluster. For jobs smaller than 18 nodes and not specifying --segment, Slurm will not fragment the allocation across block boundaries. Instead, it will keep the job queued until sufficient resources become available in the cluster.

This guarantees that all the allocated nodes for this job will be able to communicate with NVLink, providing consistent performance regardless of the current state of the cluster. For jobs larger than 18 nodes and not specifying --segment, Slurm will fit the allocation in the minimum amount of blocks required.

To configure topology/block it is recommended to rely on the file topology.yaml, a feature introduced in Slurm 25.05. For example, to define two GB200 NVL72 domains use the following script:

The Slurm topology/block plugin supports multiple levels of hierarchical grouping, where the first level would be the NVL72 domains, and the higher levels of blocks can reflect the physical reality of the compute fabric design such as the performance impact when crossing data center rows or data center halls.

For a GB200 NVL72 cluster with a full fat-tree for the compute fabric, it is recommended to use only a single level of Slurm blocks for the multinode NVLink domains, as the performance cliff for crossing leaf switches is not as steep as the performance cliff for crossing NVLink domains. The minimum schedulable unit is a single node (four GPUs).

To fully benefit from the new capability, and submit a job where the application could benefit from added topology requirements, cluster users want to submit jobs with a --segment argument that is now enabled at the cluster administration level. --segment=1 should be used when