Validate GPU Cluster Readiness Before AI Workloads Land
Source
developer.nvidia.com
Author
Michelle Horton
Date
Context
Individually healthy GPUs can still perform poorly when working together. NVIDIA's Cluster Readiness Engine tests that shared behavior by running distributed workloads across groups of machines managed by Kubernetes.
Operators supply performance thresholds, so a job can finish successfully yet fail readiness testing because it missed its target. When a group fails, the diagnostic mode divides it into smaller groups and reruns tests to narrow the suspects. This gives operators workload evidence to investigate before committing production jobs. The engine reports failures; separate monitoring and remediation policies determine whether machines are removed from service.
Key takeaways · AI-distilled
NVIDIA's core argument: a GPU cluster can pass every health check yet still fail a real job, since one degraded GPU slows a synchronous run to its worst rank and links can degrade under load. NVCRE proves readiness by running actual distributed workloads.
NVCRE's diagnose mode runs hierarchical group testing: it splits a failing node group, reruns the halves until it reaches minGroupSize, and flags groups that still fail as suspects. maxConcurrent caps parallel jobs so tests do not saturate the fabric being measured.
Test scale changes what is measured, the authors note: an NCCL test inside one NVLink domain measures NVLink bandwidth, while the same test across racks measures the scale-out fabric. NVCRE offers intra-node, intra-rack, full-scale and diagnose groupings.
Pass criteria are CEL expressions over measured metrics such as bus bandwidth or TFLOPs per GPU, and no thresholds ship by default. A run that completes but misses its target is still recorded as a failure.
NVCRE only records failed nodes and reasons; it does not cordon or taint them. NVIDIA pairs it with NVSentinel, which can turn failed certifications into health events that quarantine or drain nodes, and a later passing certification can release the taint.
articleNVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
articleReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding
articleEfficient MoE Training for Biological Foundation Models
articleBuilding an Adaptive Agentic Cybersecurity System with NVIDIA Nemotron
