aiqbench analysis
NVIDIA Snapshot addresses the critical latency of LLM cold starts by enabling the checkpointing and rapid restoration of fully initialized GPU workloads within Kubernetes. By preserving CPU and GPU memory states, the technology allows inference replicas to scale instantly during demand spikes or recover from failures in seconds rather than minutes. This open-source component directly enhances the efficiency and responsiveness of AI inference infrastructure without requiring repeated model weight loading or compilation.
Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot

From NVIDIA
Full textNVIDIA Snapshot, a new open-source component of the NVIDIA DSX AI Factory Platform, makes initialized GPU workloads reusable across inference replicas—speeding startup and recovery.
When inference demand spikes or a replica fails, available GPUs don’t translate immediately into serving capacity. Initializing an LLM workload—a process known as a cold start—can take minutes, delaying scaling and recovery.
Snapshot is a checkpoint and restore system for GPU workloads on Kubernetes. It captures a fully initialized workload—including CPU and GPU memory—and restores that state on compatible infrastructure, bringing inference capacity online in seconds instead of minutes. Figure 1 shows the startup sequence. Snapshot skips repeated workload initialization; infrastructure provisioning and container image loading remain outside the measured restore path.
Figure 1. Cold-start sequence: provision the machine, load the container image, start the inference engine, load model weights into GPU memory, compile with torch.compile, and capture NVIDIA CUDA graphs
This post explains how Snapshot checkpoints and restores GPU workloads, how platform teams can integrate it, and how restore times compare with cold starts across three inference engines.
Originally developed for NVIDIA Dynamo inference workers, Snapshot provides checkpoint and restore capabilities that other inference engines, schedulers, controllers, and platforms can integrate.
Checkpoint: Once a workload is fully initialized, capture its CPU and GPU memory as a portable artifact.
Restore: Start a new Pod from that artifact on any node with the same GPU model and driver. The process continues from where it was checkpointed, so weight loading, compilation, and graph capture are skipped.
Both operations use Kubernetes custom resources. To checkpoint a Pod that is already running, create a PodSnapshot that names it. To start a Pod from a template and checkpoint it as soon as it is ready, create a SnapshotJob instead. To restore, add the annotation nvidia.com/restore-from to a new Pod, naming the PodSnapshot to restore from. The workload only needs a small handshake to pause before the checkpoint and resume after the restore. The repository has ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM.
Snapshot can checkpoint and restore compatible workloads that span multiple GPUs on a single node as a single checkpoint artifact across ranks. This supports fast startup for large mixture-of-experts (MoE) models using parallelism strategies such as tensor parallelism (TP), data and expert parallelism (DEP), and decode context parallelism (DCP). Workloads must meet Snapshot’s checkpoint/restore and runtime compatibility requirements. See the example configurations.
Crusoe, an upstream contributor to Snapshot, reduced GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300.
Saturn Cloud, a token factory platform, benchmarked NVIDIA Snapshot for scale-from-zero inference, cutting vLLM startup from 2–8.5 minutes to 6–34 seconds on NVIDIA HGX B300.
The following benchmarks show how Snapshot accelerates inference startup across vLLM, SGLang, and NVIDIA TensorRT LLM (TRT-LLM)—with models from 600 million parameters to mixture-of-experts (MoE) models with 1 TB checkpoints on 8 GPUs. Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start.
Cold start: The workload starts from scratch. The clock runs from process launch until engine initialization returns, before inference. The measurement is divided into four phases (Table 2): Startup: Starting the engine and worker processes, importing Python libraries, and setting up communication between GPUs. Some just-in-time (JIT) kernel compilation also happens here. Model load: Reading the model weights and constructing the model’s tensors, including any engine-specific conversion or repacking done while loading.
Initialization: Preparing the loaded model to serve: memory profiling, allocating the key-value (KV) cache and runtime buffers, and other kernel and runtime preparation. Compilation, autotuning, and graph capture: Compiling kernels (for example, with torch.compile), autotuning them for the GPU, and capturing CUDA graphs.
Startup: Starting the engine and worker processes, importing Python libraries, and setting up communication between GPUs. Some just-in-time (JIT) kernel compilation also happens here.
Model load: Reading the model weights and constructing the model’s tensors, including any engine-specific conversion or repacking done while loading.
Initialization: Preparing the loaded model to serve: memory profiling, allocating the key-value (KV) cache and runtime buffers, and other kernel and runtime preparation.
Compilation, autotuning, and graph capture: Compiling kernels (for example, with torch.compile), autotuning them for the GPU, and capturing CUDA graphs.
Snapshot: The workload is restored from its checkpoint. The clock runs from the moment Snapshot is asked to restore until the workload is serving again. The measurement is divided into four stages (Table 3): Agent setup: The Snapshot agent on the node locates the checkpoint artifact and prepares the target Pod and GPUs. Checkpoint/Restore in Userspace (CRIU) restore: CRIU, the standard Linux tool for checkpointing processes, reads the process image from storage into host memory. Speed is set mostly by the storage backend. CUDA restore: Copying GPU state back into GPU memory. Wake/remap: Adapting the process to the new node and waking the engine.
Agent setup: The Snapshot agent on the node locates the checkpoint artifact and prepares the target Pod and GPUs.
Checkpoint/Restore in Userspace (CRIU) restore: CRIU, the standard Linux tool for checkpointing processes, reads the process image from storage into host memory. Speed is set mostly by the storage backend.
CUDA restore: Copying GPU state back into GPU memory.
Wake/remap: Adapting the process to the new node and waking the engine.
Both measurements exclude pulling the container image and setting up the container runtime. The KV cache release is triggered before every checkpoint, so it is not part of the artifact.
Table 1 also reports checkpoint time and size: how long it takes to capture a fully initialized replica and how much storage the stored artifact requires. This is a one-time cost per model and configuration.
NVIDIA driver: 615.71.09
Inference engines: vLLM 0.31.0 SGLang 0.5.21 TensorRT LLM 1.3.0
TensorRT LLM 1.3.0
The benchmarks show four main findings.
Restore times stayed under 43 seconds: Across all tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds—8x to 36x faster than cold start. The largest MoE models, with checkpoints of about 600 GB to 1 TB across eight GPUs, restored in under 43 seconds compared with cold-start times of 6 to 19 minutes.
Restoring skips more than weight loading: Cold-start time includes loading weights, compiling and autotuning kernels, and capturing CUDA graphs. Snapshot restores the initialized workload, skipping these repeated steps.
Checkpointing pays off with the first restored replica: In every tested configuration, checkpoint time plus restore time was shorter than a single cold start. Checkpoint creation took 15 seconds to 5.5 minutes, depending on the model and configuration.
Moving GPU state dominates large restores: For the eight-GPU MoE configurations, restoring GPU state took 18 to 29 seconds. Restoring CPU process state with CRIU took 4.2 to 10.8 seconds.
Figures 2 to 4 compare cold-start and Snapshot restore times for vLLM, SGLang, and TensorRT LLM across the tested models and parallelism configurations. Table 1 provides the data for every configuration, including checkpoint time and size. All times are in seconds. In the figures and tables, TEP8 denotes tensor and expert parallelism across eight GPUs.
Figure 2. vLLM: Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 4.7 to 39.6 seconds
Figure 3. SGLang: Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 3.6 to 42.7 seconds
Figure 4. TensorRT LLM (TRT-LLM): Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 7.79 to 33.55 seconds
DeepSeek-V4.1-Flash
Table 1. Cold start compared with a Snapshot restore on NVIDIA B200 with VAST (NFS) storage, on 1 to 8 GPUs. Checkpoint time is a one-time cost per configuration. Speedup is cold start time divided by Snapshot restore time
Table 2 breaks down the cold-start time, showing which initialization phases a restore skips.
Table 2. Where the cold start time goes, by phase, in seconds. Phase timings may differ from the total because of rounding or intervals not captured by logging and instrumentation
Table 3 breaks down restore time into agent setup, CRIU restore, CUDA restore, and wake/remap.
Table 3. Where the Snapshot restore time goes, by stage, in seconds. Together, the four stages make up the Snapshot column in Table 1
Figure 5 shows the restore-time breakdown for vLLM.
Figure 5. Snapshot restore time by stage on vLLM, shown as an example. The number above each column is the total restore time in seconds
Benchmarking tools and reproduction instructions are available in the repository.
To try Snapshot with a GPU workload on Kubernetes, start with the installation instructions and usage guides in the NVIDIA Snapshot repository.
Try it out with NVIDIA Dynamo to restore workers by following the user guide.
To discuss an integration or contribute feedback, open an issue or join the NVIDIA Dynamo Slack workspace and visit the #snapshot-dev channel.
Visit the NVIDIA booth #170 at KubeCon for Kubernetes deep dives and live demos.
Full text under the publisher's open licence · 1475 words
Other coverage of this story
Related coverage
Readie: Low-Latency Code Sandboxes for AI Agents
arXiv · Nov 18, 2026

NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes
NVIDIA · May 27, 2026
State-Aware Activation Recomputation with Replay Contracts and Selective Preservation
arXiv · Nov 4, 2026

NVIDIA Expands Open Source Commitment With OpenShell Contribution and New AI Infrastructure
NVIDIA · Nov 10, 2026