Enforce GPU memory limits in Kubernetes with CUDA 13.4 and KAI Scheduler

NVIDIA has released CUDA 13.4, introducing transparent memory boundary enforcement for GPU workloads. To operationalize this in Kubernetes, the open-source GPU Fractioning Operator now works with KAI Scheduler v0.18.0 to manage fractional GPU resources. This combination allows platform teams to share GPUs more efficiently while maintaining strict memory isolation.

What happened

Many AI inference and development workloads do not require an entire GPU. Assigning full devices to small jobs leads to significant idle capacity. While NVIDIA GPUs have long supported sharing via Multi-instance GPU (MIG), time-slicing, or Multi-Process Service (MPS), previous methods lacked robust memory enforcement in shared modes. Time-slicing allowed processes to take turns but did not prevent one process from consuming all available memory. MPS allowed concurrent execution but relied on application-level discipline for memory usage.

CUDA 13.4 addresses this gap by enforcing memory boundaries transparently at the driver level. Applications see only their allocated fraction of memory, and allocations exceeding this limit fail immediately. To bridge the gap between this low-level capability and Kubernetes orchestration, NVIDIA introduced the GPU Fractioning Operator. Hosted under the KAI Scheduler organization on GitHub, this operator runs alongside the NVIDIA GPU Operator. It manages the setup of MPS on every node, applies per-workload memory limits, and reports usage per fraction rather than per whole GPU.

KAI Scheduler v0.18.0 integrates directly with this operator through a new GPU-sharing mode called NvFractions. The scheduler selects appropriate GPUs based on available memory and writes specific annotations to Pod specifications. The GPU Fractioning Operator reads these annotations and configures the node to enforce the requested memory limits and compute modes before the container starts. This separation of concerns ensures that scheduling decisions remain distinct from node-level enforcement.

How it works

The system relies on two compute modes for sharing GPU resources: time-slicing and sm-sharing. In time-slicing, workloads take turns using the entire GPU. If one workload is idle, its time slot is redistributed to others. This mode provides full error isolation, meaning a crash in one workload does not affect others. In sm-sharing, the GPU’s streaming multiprocessors (SMs) are divided among concurrent workloads. All active workloads run simultaneously, and unused SMs are dynamically reallocated. However, sm-sharing places workloads in a single error domain, so a fault in one can impact others.

CUDA 13.4 enforces memory limits regardless of the compute mode. A workload can request an exact memory amount or a range consisting of a guaranteed minimum and a burstable maximum. If a workload attempts to allocate memory beyond its limit, the allocation fails just as it would on a smaller physical GPU. This enforcement is transparent to the application, requiring no code changes. The GPU Fractioning Operator ensures these limits are applied by configuring the underlying MPS daemons and monitoring usage per fraction.

Schedulers communicate requirements to the operator using standard Kubernetes Pod annotations. These annotations specify the guaranteed memory request, the optional burst limit, the desired compute mode, and the specific GPU devices assigned. The operator parses these fields and translates them into node-level configurations. This mechanism allows any Kubernetes scheduler capable of writing these annotations to leverage CUDA’s memory enforcement, though KAI Scheduler is the first to offer native integration.

Key details

• CUDA 13.4 introduces transparent memory boundary enforcement, allowing exact memory amounts or burstable ranges for shared GPU workloads.
• The GPU Fractioning Operator is an open-source tool that sets up MPS, applies memory limits, and measures usage per GPU fraction.
• KAI Scheduler v0.18.0 includes an NvFractions mode that writes necessary Pod annotations for the operator to enforce.
• Workloads can choose between time-slicing (serial execution with error isolation) and sm-sharing (parallel execution with dynamic SM allocation).
• Memory requests can be defined as absolute sizes (e.g., "40Gi") or as portions of a GPU (e.g., "0.5").
• Administrators enable the integration during KAI Scheduler installation using a Helm flag, which installs the GPU Fractioning Operator as a dependency.

Why it matters

For engineering teams building ML infrastructure, this update significantly improves resource utilization without sacrificing stability. Previously, sharing GPUs often meant choosing between poor isolation (MPS) or rigid partitioning (MIG). With CUDA 13.4 and the GPU Fractioning Operator, teams can pack more workloads onto existing hardware while ensuring that no single job can starve others of memory. This is particularly valuable for inference services and development environments where workloads vary in size and intensity.

The ability to define burstable memory limits adds flexibility for batch jobs or training tasks that may occasionally need more resources. If memory is free, a workload can expand into it; if not, it stays within its guaranteed minimum. This dynamic allocation reduces the need for over-provisioning. Furthermore, the clear distinction between time-slicing and sm-sharing allows architects to match the sharing mode to the workload’s fault tolerance requirements. Critical production services can use time-slicing for isolation, while resilient batch jobs can use sm-sharing for higher throughput.

What you can do

• Install KAI Scheduler v0.18.0 or later with the global.nvFractions.set=true Helm flag to enable GPU fractioning support.
• Review the GPU Fractioning Operator repository on GitHub for installation guides and configuration details.
• Update Pod specifications to include annotations for memory requests, limits, and compute modes when deploying fractional workloads.
• Choose time-slicing for workloads requiring error isolation and sm-sharing for those needing parallel execution and dynamic SM usage.
• Monitor GPU usage per fraction using the metrics provided by the GPU Fractioning Operator to optimize resource allocation.
• Join the KAI Scheduler Slack channel or bi-weekly community meetings to ask questions and contribute to the project.