Agent Substrate: Advancing Kubernetes for Agents at Scale

Agent Substrate is an open-source runtime for running large numbers of AI agents on Kubernetes, the widely used system for scheduling containers across a cluster. In a post on its developer blog, NVIDIA says the project has been donated to the Cloud Native Computing Foundation as a Sandbox project, describes the code it has contributed, and explains how Substrate is meant to work alongside its own inference software. The problem Substrate addresses is that an agent session spends most of its life waiting for a person, a tool or a model response. Kubernetes gives a platform team two poor options for that pattern: keep one Pod (Kubernetes's unit of running containers) warm per session and pay for memory that sits idle, or create a Pod for each step and pay a cold start of several seconds every time, losing the agent's state in between.

Substrate takes a third route. It keeps a small pool of ordinary, already-running worker Pods and maps a much larger set of agents onto them. When an agent goes idle, Substrate snapshots it out of its sandbox, memory and filesystem included, and frees the worker. When the next request arrives, the agent is restored onto whichever worker is free, so waking it skips the Kubernetes API server, scheduler and node setup that make a new Pod slow. Every agent runs inside an isolation layer, either gVisor or a lightweight virtual machine, and the project README says plain Linux containers are not supported. The project claims 10x higher density than standard containers and resumes in under 500 milliseconds; neither NVIDIA's post nor the README shows the benchmark behind those figures.

NVIDIA's own contributions are smaller pieces built for three workloads it names: coding assistants, reinforcement-learning training that fires off bursts of sandboxed environments, and short-lived GPU sandboxes for researchers. Request parking makes a request wait, with a bounded timeout and a cap on the queue, when every worker is busy, where Substrate previously returned an error. Per-container limits let a microVM sandbox stop one runaway container without killing the agent beside it. GPU access is still work in progress: NVIDIA describes device support built on Kubernetes Dynamic Resource Allocation, and only proposes a design for suspending a GPU sandbox, because a saved GPU state can usually be restored only onto the same GPU model and sometimes only with the same driver. Under that design, a snapshot records the device properties it depends on and is restored onto a matching device, or the sandbox starts cold if none is free.

The second half of the post concerns NVIDIA Dynamo, NVIDIA's open-source inference server, which would sit between agents and the GPUs running the model. Substrate would pass Dynamo hints such as session identity and tool-call boundaries, so Dynamo can send an agent's next model call to the GPU that already holds its cached context. Dynamo would return estimates such as time to first token and how saturated the GPU pool is, so Substrate can suspend an agent during a long model wait and wake it as the response nears completion. NVIDIA calls this the most important part of the integration, and also says parts of it are experimental and that GPU support is being rebuilt. The supported conclusion is that Substrate's core multiplexing exists today, while the GPU and Dynamo coordination is a stated direction with no measurements yet.

The GPU suspend problem in this post is close to the one addressed by NVIDIA Snapshot, a checkpoint and restore system for GPU inference workloads that NVIDIA described the same day. Snapshot also restores only onto the same GPU model and driver, and neither post says whether Substrate's GPU design will use Snapshot.