Reducing container cold start times using SOCI index on DLAMI and DLC

Deep Learning AMI and AWS Deep Learning Containers are now enabled with support for SOCI snapshotter and index. Seekable OCI (SOCI) is a technology that enables efficient container image management through selective file downloading. It uses a layer-based indexing system to map file locations within container images, allowing containers to start with only the necessary files loaded (lazy loading). This approach reduces network bandwidth usage and improves container startup times, making it particularly valuable for organizations managing large container images in cloud environments.

In this post, we look at how to use SOCI on publicly available Deep Learning AMIs and Containers, when to use the various SOCI modes provided by the tool, and how to quickly and efficiently use this tool in your workloads today.

As organizations deploy artificial intelligence (AI) and machine learning (ML) workloads at scale, container startup time has become a bottleneck in production environments. Whether it’s spinning up training jobs, serving inference endpoints, or scaling GPU clusters automatically, the time spent downloading multi-gigabyte container images directly impacts cost, user experience, and operational efficiency. Traditional container deployment approaches force teams to download entire images before workloads can begin. This process can take multiple minutes to start up images commonly used in production. During development, a few minutes of wait time is barely noticeable. In production, those same minutes add up fast.

Prolonged cold start times. Standard Docker image pulls of 15–20 GB can take 4–6 minutes per instance, delaying training jobs and inference endpoints during scaling events.

Wasted compute resources. GPU instances sit idle during image pulls, burning through expensive compute hours while waiting for container initialization to finish.

Scaling bottlenecks. When demand spikes trigger automatic scaling, slow container startup times prevent rapid response, leading to degraded performance or dropped requests.

Bandwidth constraints. Large-scale deployments pulling massive images simultaneously can saturate network bandwidth, creating cascading delays across the infrastructure.

Developer productivity. Data scientists and ML engineers waste valuable time waiting for containers to start during iterative development and experimentation cycles.

When pulling a container for your workloads, AWS Deep Learning AMIs (DLAMI) and Deep Learning Containers offer three options: the standard Docker pull, SOCI parallel pull, and SOCI lazy loading through SOCI index. Think of these as a sliding scale of tradeoffs. Docker pulls are sequential and slow. SOCI parallel pull provides faster startup times by chunking downloads at the cost of compute resources. SOCI lazy loading provides near-instant container loading but requires files to be fetched on demand. You can use the following guide to choose the right mechanism for your workloads:

The choice between lazy loading and parallel pull modes depends on the image, instance specifications, and storage configuration. Lazy loading requires images to have a SOCI index. Without one, the system falls back to standard pulling.

Lower-spec instances should use lazy loading to conserve resources, while high-spec instances with multiple vCPUs and high network bandwidth benefit from parallel pull mode. Storage performance varies: EBS volumes are bounded by their provisioned IOPS and volume type, potentially creating bottlenecks during unpacking, while NVMe instance store delivers maximum I/O performance at the cost of data persistence across instance stop/start cycles.

The following diagram shows the architecture for using SOCI with DLAMI and Deep Learning Containers.

The following benchmarks compare standard Docker pulls against SOCI snapshotter in both lazy loading and parallel pull modes.

Lazy loading mode starts containers immediately by fetching only the necessary data on demand, with remaining layers loaded in the background as needed.

Important: Lazy loading mode requires the container image to have a SOCI index stored in the registry. Without a SOCI index, the snapshotter will fall back to standard pull behavior, and you won’t see any performance improvement. AWS Deep Learning Containers (DLCs) with the -soci tag suffix come with SOCI indexes pre-created and pushed to the registry, enabling lazy loading out of the box. For custom images, you must create and push SOCI indexes

Instance Type: g5.2xlarge

AMI: Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) 20260413 (ami-06abbbf2049359343)

Docker Image: public.ecr.aws/deep-learning-containers/vllm:0.19.0-gpu-py312-ec2-soci

Image Size: 9.72GB (compressed), 32.7GB (disk usage)

We use Docker to start the inference server directly. Since no image exists locally, Docker pulls and extracts the entire image before starting the container.

Total time: 6m59.099s.

We use nerdctl with SOCI snapshotter to start the inference container. Although no image exists locally, the SOCI-indexed image allows nerdctl to pull only the index and necessary layers to start the container, enabling lazy loading of remaining layers. Total time: 21.125s.

Using SOCI snapshotter with lazy loading, the container started in 21.125 seconds, compared to 6 minutes 59.099 seconds with standard Docker. This improvement is achieved because SOCI pulls only the necessary layers to start the container, with remaining layers loaded on demand as needed.

While lazy loading mode starts containers immediately by fetching only the required data on-demand, parallel pull mode downloads the entire image before startup but does so with higher concurrency than standard Docker pulls. This mode is ideal when you need the full image available at startup or when running I/O-intensive workloads.

Instance Type: g5.4xlarge

AMI: Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) 20260413 (ami-06abbbf2049359343)

Docker Image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker

Image Size: 19.32GB (compressed), 60.4GB (Disk Usage)

Note: We use a private ECR image for this benchmark because public ECR is fronted by Amazon CloudFront, which limits network bandwidth and affects parallel mode performance. Private ECR is served directly from Amazon Simple Storage Service (Amazon S3), providing higher throughput.

The SOCI snapshotter on Deep Learning AMI defaults to lazy loading mode. To enable parallel pull mode, modify the configuration file at /etc/soci-snapshotter-grpc/config.toml:

Tip: You can tune max_concurrent_downloads_per_image and max_concurrent_unpacks_per_image based on your instance type and network bandwidth. For detailed tuning guidance, see Introducing Seekable OCI Parallel Pull Mode for Amazon EKS.

Standard Docker pull downloads and extracts layers with limited concurrency.

Total time: 4m 44.163s

Using nerdctl with SOCI parallel pull mode uses increased concurrency for both downloads and unpacking operations.

Total time: 2m 12.846s

Using SOCI parallel pull mode reduced image pull time from 4 minutes 44 seconds to 2 minutes 12 seconds, representing a 2.2x improvement in pull performance.

Parallel pull mode — Achieved a 2.2x improvement in image pull time (from 4 minutes 44 seconds to 2 minutes 12 seconds)

Choose lazy loading mode when you need the fastest possible container startup, or parallel pull mode when you need the full image available before your workload begins.

If you launched EC2 instances to test SOCI snapshotter, terminate them to avoid incurring ongoing charges. Delete any container images you pushed to Amazon Elastic Container Registry (Amazon ECR) during testing, and remove any SOCI indexes you no longer need.

DLAMI and Deep Learning Containers are publicly available today with SOCI snapshotter and SOCI index. For more information on publicly available DLAMI and Deep Learning Containers, you can check out SOCI Index DLAMI to select the images that support SOCI, and check out the Deep Learning Container repository to get more information on supported images with SOCI index.

For detailed configuration guidance and best practices, refer to the SOCI documentation and the Deep Learning Container SOCI documentation.