Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

NVIDIA has released an open-source Kubernetes operator that lets several workloads share one GPU with a hard cap on how much GPU memory each can use. Many AI jobs need only part of a GPU, and giving each a whole device leaves capacity idle. The older ways to share had gaps: hardware partitioning (MIG) splits a GPU into fixed isolated instances, and time-slicing lets processes take turns but does not enforce memory limits, so one job can take memory its neighbors need. The NVIDIA post explains how the pieces fit.

CUDA 13.4 supplies the enforcement. A workload can ask for an exact memory amount, and an allocation beyond it fails as it would on a smaller GPU, or it can get a guaranteed minimum plus a maximum to grow into when memory is free. The CUDA 13.4 release described this as part of a new version of Multi-Process Service, NVIDIA's mechanism for running several processes on one GPU. But a Kubernetes cluster cannot use it on its own. The new GPU Fractioning Operator does the node-level work: it runs Multi-Process Service on each GPU node, applies each container's memory limits and compute mode before the process starts, and reports usage per fraction instead of per GPU.

The scheduler and the operator split the job through annotations on the Pod, the Kubernetes unit that runs containers. The scheduler picks a GPU with enough free memory and writes the assigned devices, memory request, limit and compute mode into those annotations; the operator reads them and enforces them. NVIDIA's KAI Scheduler, from version 0.18.0, is the first scheduler to do this, through a mode called NvFractions, and the post says any scheduler that writes the same annotations can use the operator.

The compute mode is the main choice a user makes. Time-slicing, the default, has workloads take turns on the whole GPU, and each runs in its own error domain, so a crash in one does not affect the others. SM-sharing splits the GPU's processing units so all workloads run at once, which avoids waiting but puts them in a single error domain; NVIDIA recommends it only for workloads you trust to run together. A GPU runs one mode at a time, so KAI Scheduler never mixes them on the same device. The operator is one of the components NVIDIA listed when it open-sourced the Run:ai platform in October.