Control How Your GPU Shares Work with Green Contexts

GPU applications increasingly consist of multiple independent components running at the same time within a single process: a latency-sensitive operator alongside a throughput-oriented background kernel; a data preprocessing stage alongside model inference; or multiple stages of a processing workflow sharing a single GPU.

Controlling how GPU resources are shared between them remains difficult. Components can interfere unpredictably, and existing tools offer limited ability to partition resources.

Green contexts address this by letting applications explicitly select a subset of GPU execution resources and target work to those resources directly. Traditional CUDA contexts weren’t designed for this usage model. They are heavyweight, incur hardware context-switch overhead, and reflect assumptions from an earlier era, when GPUs were smaller and applications typically ran as a single dominant workload.

Green contexts have been available in the Driver API since NVIDIA CUDA 12.4. Starting with CUDA 13.1, they are also accessible through the Runtime API, allowing an application, within its process, to explicitly define where work runs and how execution resources are divided.

One primary use of green contexts is SM partitioning. By assigning a specific subset of SMs to a green context, applications can target work submitted through that green context to those SMs. This can enable multiple workloads to run concurrently on the GPU without competing for the same compute units.

In addition to SM partitioning, green contexts can provision workqueue resources. In the traditional model, independent stream-ordered workloads may map to the same underlying workqueues, introducing unintended serialization even when sufficient execution resources are available. By provisioning workqueues explicitly, green contexts allow applications to express expected concurrency and reduce false dependencies.

Green contexts are lightweight to create and destroy, and creating or destroying one doesn’t implicitly synchronize unrelated GPU work. They also provide a more explicit programming model where applications target work to a chosen green context rather than relying only on implicit/current device state.

Historically, CUDA Runtime applications typically selected a device with cudaSetDevice(), created streams, and submitted work to those streams. The stream’s execution target was inferred from the current device or context for the calling thread at the time the stream was created.

Green contexts make that targeting more explicit. An application creates a green context for a chosen set of resources, then creates streams from that green context. Work submitted to those streams is associated with the green context’s resources.

In the CUDA Runtime API, green contexts are represented with the cudaExecutionContext_t type, a Runtime abstraction for CUDA contexts. For green-context use, the important point is that cudaGreenCtxCreate() returns a handle that can be passed directly to APIs such as cudaExecutionCtxStreamCreate(), instead of relying on implicit thread-local device or context state.

For streams, this shift is reflected directly in how applications create them.

Note: For brevity, the following code snippets omit full error checking. Production code should check all CUDA Runtime API return values, use cudaGetLastError() or cudaPeekAtLastError() after kernel launches, and check synchronization calls such as cudaStreamSynchronize() for asynchronous execution errors.

The code changes are minimal, but the mental model is clearer. Instead of relying on hidden thread-local device state, applications explicitly choose a green context and create streams for it. The rest of the application code can continue using the same stream-based CUDA programming model.

Applications that want to target the full device can continue using the traditional Runtime model. For APIs that take an explicit context handle, they can obtain the device-wide context with cudaDeviceGetExecutionCtx().

GPU workloads may often require a small, latency-sensitive kernel to share a device with a larger throughput-oriented worker. A canonical example is communication/GEMM overlap in distributed training and inference, or latency-sensitive operators in AI sensor processing platforms like NVIDIA Holoscan that require starting as soon as possible.

One of the standard tools for prioritizing such critical work is CUDA stream priority. Unfortunately, setting priority isn’t enough to guarantee immediate execution of a higher priority kernel, when the bulk kernel fully occupies all of the GPU’s SMs. If bulk kernels are queued up and the critical kernel arrives, the scheduler hands critical kernel threads blocks to the next SM(s) that free(s) up. Stream priorities can’t preempt a block that’s already executing on an SM, so it needs to wait for some of the blocks to drain.

Let’s create a sample and look at some results. The setup code is the new piece we need to write:

The stream carries the partition and the kernel sees only the SMs it’s allowed to run on.

The critical kernel is a tiny integer workload. The bulk kernel is 20 back-to-back launches of a 4M-thread sqrtf loop designed to saturate the device. We measure the wall-clock latency of the critical kernel while bulk is running.

Table 1. Critical kernel latency by execution mode

Stream priority alone is a huge win over not using it. Roughly 27x faster than equal-priority streams competing for the same SMs. But priority has a ceiling. An additional 20x is wasted waiting for bulk blocks to drain, even with the GPU scheduler prioritizing the critical kernel. Green contexts skip that wait entirely because the critical SMs are dedicated to that workload. The tradeoff is that less SMs are available for the bulk kernel’s execution.

The same skeleton can be used to achieve desired concurrency rather than latency improvement, such as overlapping a communication kernel with a GEMM kernel so both make progress. Advanced use-cases often use both green context partitions and stream priorities to achieve desired performance targets.

Runs multiple independent workloads concurrently on the same GPU.

Build your application with CUDA 13.1 or newer.

Query device resources and create a green context for the resource partition you want to target.

Create streams for that green context and submit work using the same stream-based CUDA programming model.

For APIs that take an explicit context handle and should target the full device, use cudaDeviceGetExecutionCtx().

Green contexts are opt-in and additive. Existing applications can continue to target the full device unchanged and adopt green contexts incrementally where finer control over execution is needed.

For additional details and examples, see the CUDA Programming Guide and CUDA Runtime API documentation, and try green contexts today.