Build a Validated Kubernetes Runtime for NVIDIA Dynamo
Source
developer.nvidia.com
Author
Stefan Schimanski
Date
Why it matters
Explains how cache-aware routing works with Dynamo on Kubernetes and how to choose between the native frontend router and the Gateway API path, with a reproducible deployment flow.
Key takeaways · AI-distilled
NVIDIA AI Cluster Runtime (AICR) packages Dynamo with its Kubernetes dependencies, including GPU Operator, the DRA GPU driver, KAI Scheduler, Grove, NVSentinel and Prometheus, as recipes with pinned versions, checksums and validation checks.
In Dynamo 1.4, components discover each other through Kubernetes resources, so AICR's default path no longer uses NATS and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in.
Dynamo offers two cache-aware routing paths: its Frontend router in kv mode, the smallest and most production-proven topology, or the Gateway API inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → Extension with a Dynamo Endpoint Picker, which suits shared ingress where the gateway owns timeouts, retries and fallback.
Both paths route on KV-cache events that vLLM workers publish over ZMQ. NVIDIA notes that predicting cache state from past routing cannot see evictions, so the router's view can drift and routing quality can degrade under memory pressure.
AICR's flow runs snapshot, recipe, bundle, deploy (Helmfile, Helm or GitOps) and validate, and the application's DynamoGraphDeployment is installed only after deployment and conformance checks pass. Performance validation runs separately because it consumes the GPUs under test.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
