Spur: Modern GPU Job Scheduling for HPC and AI Workloads

13 min read. | 3296 total words.

The explosion of AI and large-scale model training has fundamentally changed how organizations think about GPU clusters. Traditional HPC schedulers were built in an era when CPUs dominated, and GPU support was retrofitted as an afterthought. Meanwhile, Kubernetes emerged from the cloud-native world with powerful orchestration primitives—but batch GPU scheduling and Slurm-style job workflows remain friction points unless you assemble a full ecosystem of operators, custom schedulers, and queueing layers on top of the default control plane.

We’re introducing Spur and Spur-Cloud—a modern, GPU-first job scheduler written in Rust, developed by AMD and open-sourced under Apache 2.0 as part of AMD’s Open Ecosystem initiative. Spur provides Slurm-compatible CLI and APIs (with broader parity actively under development) while delivering features designed for today’s AI infrastructure: topology-aware GPU scheduling, embedded high availability via Raft consensus, native Kubernetes integration, and vendor-agnostic device management through the Container Device Interface (CDI).

In this post, we’ll explore how Spur addresses operational pain points in GPU cluster management, demonstrate its core capabilities with hands-on examples, and show how Spur-Cloud extends the platform into a complete GPU-as-a-Service solution.

Managing GPU clusters at scale presents unique challenges. Training a large language model might require 64 MI300X GPUs with XGMI interconnect topology, scheduled across 8 nodes with precise locality constraints. A computer vision pipeline might need fractional GPU allocation—8 independent jobs sharing a single 8-GPU node. Inference workloads demand rapid placement and teardown across heterogeneous hardware (MI300X, MI250, A100). Traditional schedulers struggle with this complexity.

Consider a common scenario: you submit a multi-node training job requesting --gres=gpu:mi300x:8 across 4 nodes. Your scheduler must:

Spur was built from the ground up to handle these requirements with first-class GPU support, not as an afterthought.

Why a New Scheduler?#

Traditional HPC schedulers proved the batch job model over decades, but architectures designed for the 2000s carry operational overhead that modern infrastructure shouldn’t require: external database dependencies for accounting, shared state management for HA, C codebases accumulating memory-safety CVEs, and plugin-based GPU support that requires careful flag combinations rather than sensible defaults.

This flexibility is valuable for organizations running diverse workloads, but for GPU-focused clusters, the complexity becomes overhead. Default K8s behavior requires explicit podAffinity and topologySpreadConstraints to guarantee GPU locality—possible, but not automatic.

Spur asks: what if we rebuilt HPC batch scheduling with 2026 technology and GPU workloads as the primary use case?

The result: operational simplicity without sacrificing the batch job model that HPC users rely on.

spur-cli — Multi-call binary providing sbatch, squeue, scancel, etc., for CLI compatibility

spur-cli — Multi-call binary providing sbatch, squeue, scancel, etc., for CLI compatibility

State and persistence: Cluster scheduling state—jobs, nodes, partitions, and the queue—is replicated via embedded Raft consensus in spurctld. No external database is required to submit, schedule, and run jobs. Spur-Cloud adds its own PostgreSQL database for billing and session metadata.

1. GPU-First Scheduling with Topology Awareness#

Spur’s backfill scheduler understands GPU topology through CDI device annotations and system topology detection. When you request GPUs, the scheduler considers:

The default Kubernetes scheduler treats GPUs as opaque extended resources and does not natively understand XGMI/NVLink locality—but production K8s clusters often add topology awareness via schedulers like Volcano, Yunikorn, or Kueue, and Operators with Dynamic Resource Allocation (DRA). Spur integrates topology-aware batch scheduling into the controller itself, without requiring a separate scheduler stack for HPC-style jobs.

2. Vendor-Agnostic Device Management via CDI#

Spur uses the Container Device Interface (CDI) for GPU device management, eliminating hardcoded vendor logic. CDI specifications describe how to inject devices, including AMD ROCm GPUs and other vendor accelerators, into containers.

On startup, spurd reads CDI specs from /etc/cdi/ and /var/run/cdi/, falling back to auto-discovery via sysfs for AMD GPUs if no specs exist. Administrators can provide custom CDI specs for exotic hardware without modifying Spur.

3. Built-in High Availability with Raft Consensus#

Traditional HPC schedulers rely on external databases (MySQL, MariaDB) for accounting state and often use manual failover for the controller. Spur embeds Raft consensus directly in spurctld for scheduling state, providing:

This is not a novel consensus algorithm; etcd and Spur both rely on Raft. The difference is deployment model: Spur co-locates scheduling state with the controller binary, which reduces moving parts for bare-metal and batch-focused clusters.

The corresponding config (spur.conf):

The Raft node ID is auto-detected from the StatefulSet pod ordinal (spurctld-0 → node 1, spurctld-1 → node 2). On leader failure, the remaining nodes elect a new leader within ~1-3 seconds, and job submissions resume automatically.

4. Native Kubernetes Integration#

Spur runs jobs as native processes on bare metal or VMs for maximum performance, but it also integrates with Kubernetes for hybrid deployments. The spur-k8s-operator watches SpurJob custom resources and submits them to spurctld:

The operator creates a Kubernetes Pod with GPU resource requests (amd.com/gpu: 1) on the allocated node. This provides:

Where Spur adds value over the default kube-scheduler is multi-node batch coordination: a job requesting 4 nodes with 8 GPUs each needs gang scheduling and topology constraints that vanilla Pod scheduling does not express natively. A single Pod requesting nvidia.com/gpu: 8 (or amd.com/gpu: 8) is typically packed onto one node by Kubernetes—but coordinating a 32-GPU job across 4 nodes with locality constraints is where dedicated batch schedulers (Volcano, Kueue) or Spur’s integrated approach matter. Spur targets teams that want Slurm-compatible job semantics on top of (or alongside) their K8s cluster.

5. Rootless Containerized Workloads#

Spur includes a built-in container runtime for running unprivileged containers without requiring site-wide installation of Singularity, Enroot, or Apptainer—common requirements in Slurm environments. Kubernetes clusters can also run rootless workloads via containerd user namespaces, CRI-O rootless mode, and related runtimes; Spur’s advantage is shipping an integrated, HPC-oriented runtime in the agent itself.

Inside the container, GPU devices are automatically available via ROCR_VISIBLE_DEVICES. The container filesystem is read-only by default (overlayfs writeable layer optional), and the job’s home directory is bind-mounted for persistent storage.

This eliminates the “works on my machine” problem—users package dependencies in container images, and admins don’t need to install CUDA/ROCm toolchains on every node.

6. WireGuard Mesh Networking for Multi-Site Clusters#

Cloud bursting and multi-datacenter GPU clusters require secure, low-latency communication across network boundaries. Spur includes integrated WireGuard mesh networking to create a unified overlay network for job communication—configured in the scheduler stack rather than as a separate CNI or VPN project.

Kubernetes clusters can achieve similar encryption and multi-cluster connectivity via CNI plugins (Cilium and Calico support WireGuard) and tools like Submariner for multi-cluster networking. Spur’s mesh is aimed at batch jobs spanning on-prem and cloud nodes: agents auto-register WireGuard endpoints and multi-node jobs communicate over the encrypted overlay without per-job VPN setup.

Configure the mesh in spur.conf:

Agents auto-detect their WireGuard IP and self-report it during registration. Multi-node jobs communicate over the mesh without exposing public IPs—when a job runs across on-prem and cloud nodes, MPI or NCCL communication happens over the encrypted WireGuard tunnel, transparent to the application.

7. Slurm Compatibility: Migration Path#

Organizations with existing Slurm workflows can migrate incrementally. Spur provides a growing compatibility surface:

C FFI shim — libspur_compat.so provides ABI compatibility for Slurm-linked binaries

C FFI shim — libspur_compat.so provides ABI compatibility for Slurm-linked binaries

Under the hood, sbatch is the spur-cli binary invoked via a symlink. It parses Slurm batch script directives (#SBATCH), converts them to gRPC SubmitJobRequest, and talks to spurctld on port 6817. Sites with complex Slurm plugin ecosystems should plan a phased migration—see the Roadmap for ongoing parity work.

Slurm has served the HPC community for two decades and remains the de facto standard. It offers unmatched maturity, a vast ecosystem of plugins, and deep integration with HPC site management tools. Spur doesn’t aim to replace Slurm wholesale—instead, it targets operational pain points where a modern architecture provides clear advantages:

Spur’s Slurm compatibility layer enables brownfield migration—start by running Spur alongside existing infrastructure, migrate workloads incrementally, and cut over when ready.

Spur vs. a Full Kubernetes GPU Stack#

This post focuses on Spur’s strengths; a production Kubernetes GPU cluster is rarely “vanilla K8s” alone. The table below compares Spur to the ecosystem teams typically deploy—not just the default scheduler:

Complementary, not competing: AMD’s Open Ecosystem includes Operators (GPU, Network, and more) that bring device-aware scheduling, DRA, and topology awareness to Kubernetes—deployed at scale across AI and HPC infrastructure. Spur addresses a different need—Slurm-compatible batch scheduling with an opinionated, integrated stack for HPC and AI training clusters. Many organizations will run both: Kubernetes with Operators for services and cloud-native workloads, Spur for batch GPU jobs (bare metal, VMs, or via the SpurJob CRD).

Choose Spur when you want a unified batch scheduler with Slurm migration paths and integrated HA, containers, and mesh networking. Choose a full K8s GPU stack when your workloads are already Pod-native and you have invested in Volcano/Kueue, GPU operators, and cloud-native tooling.

While Spur handles job scheduling, Spur-Cloud extends it into a complete GPU-as-a-Service platform with web UI, billing, and multi-tenancy. It provides:

Spur-Cloud submits a job to spurctld via gRPC, the scheduler places it, and the K8s operator creates a Pod. The web UI provides:

Integrated terminal (xterm.js over WebSocket via kubectl exec)

Integrated terminal (xterm.js over WebSocket via kubectl exec)

Usage events are emitted when sessions start, stop, or hit time limits. Admins can export usage records for integration with accounting systems or apply their own pricing models for chargebacks.

All methods produce a platform JWT used for API authorization. Sessions are isolated per-user—users see only their own jobs and usage data (admins see all). Quotas are enforced as concurrent GPU limits (e.g., max 8 GPUs at once) to prevent resource monopolization.

Spur-Cloud leverages Spur’s scheduler-level fractional GPU allocation for cost efficiency—assigning whole GPUs to jobs via gres without requiring hardware partitioning configurations. Kubernetes can also fractionalize GPUs via hardware partitioning and device plugins; Spur’s model is simpler for multi-tenant batch and session workloads on bare-metal-style clusters.

A single MI300X node with 8 GPUs can serve multiple concurrent sessions, each isolated via ROCR_VISIBLE_DEVICES. For example:

The remaining GPUs (5-7) stay available for additional sessions. This maximizes utilization—no need to over-provision nodes or leave GPUs idle.

Try Spur locally in under 5 minutes. Step-by-step instructions are in the Quickstart guide:

For a production GPU cluster, see the deployment guide and the native-host reference. A multi-node startup playbook and Kubernetes Helm chart are landing in the repository to simplify first production deployments—check the deployment docs for the latest scripts and charts.

Configure /etc/spur/spur.conf with partition and node definitions:

Configure /etc/spur/spur.conf with partition and node definitions:

For Kubernetes, see the Kubernetes deployment guide.

Access the UI at http://localhost:5173, create an account, and launch your first GPU session.

For Kubernetes deployment, manifests are in deploy/k8s/. The platform integrates with your existing K8s cluster—no separate infrastructure needed.

The scenarios below illustrate how Spur fits common GPU cluster patterns—they are representative workflows, not references to specific production deployments.

Consider a team training a large model across 128 MI300X GPUs (16 nodes, 8 GPUs each). Key requirements:

The --constraint=xgmi ensures only nodes with XGMI interconnect are selected. Node failures during a long-running job are surfaced to operators via agent health reporting; automatic checkpoint resume and gang re-queue behaviors are on the Roadmap.

A startup offers a fine-tuning API where customers upload datasets and train custom models. Each customer job needs isolated GPU resources without over-provisioning.

Per-user quotas (e.g., max 4 concurrent GPUs per user)

Per-user quotas (e.g., max 4 concurrent GPUs per user)

Customers interact via web UI (no CLI knowledge required), and sessions auto-terminate after a configurable timeout to prevent runaway usage.

An autonomous vehicle company runs daily simulation workloads on-prem but bursts to AWS for peak demand. Spur’s WireGuard mesh connects on-prem nodes (10.44.0.0/24) and cloud instances (10.44.1.0/24) into a unified cluster.

Jobs submitted without --constraint can land anywhere. Jobs requiring low-latency data access use --constraint=datacenter to stay on-prem.

Spur is under active development by AMD and the open-source community. Upcoming features include:

As part of AMD’s Open Ecosystem initiative, Spur is fully open-source (Apache 2.0) and welcomes contributions:

Whether you’re managing a 10-node lab cluster or a 1000-GPU datacenter, we’d love to hear your feedback and see how Spur fits your workflow. AMD backs this project as part of our broader open ecosystem—alongside ROCm, Operators deployed at scale, and cloud-native tooling.

In this blog, you explored how Spur reimagines GPU job scheduling for the AI era. You saw why traditional HPC schedulers and full Kubernetes GPU stacks fall short for GPU-first batch workloads, and how Spur closes that gap with topology-aware GPU scheduling, vendor-agnostic device management through CDI, embedded high availability via Raft consensus, a built-in rootless container runtime, and WireGuard mesh networking — all behind a Slurm-compatible CLI and API for incremental migration. You also walked through hands-on examples spanning single-node quickstart, multi-node LLM training, Kubernetes integration via the SpurJob CRD, and how Spur-Cloud layers on a web UI, billing, and multi-tenancy to deliver GPU-as-a-Service.

This is only the beginning. Our team is actively expanding Spur toward broader Slurm parity, hardware-level GPU partitioning (SR-IOV), advanced preemption, multi-cluster federation, and richer observability — which we’ll cover in future series posts. Whether you’re managing a 10-node lab cluster or a 1000-GPU datacenter, we’d love to hear your feedback and see how Spur fits your workflow. We invite you to try Spur on your own cluster, join the discussion, and help shape where the project goes next.

GPU scheduling doesn’t have to be painful. As an AMD-backed, fully open-source project built on AMD’s Open Ecosystem principles, Spur is designed to work across your entire GPU infrastructure stack. Get started, explore the docs, and join the community on GitHub.

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, Instinct, EPYC, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved