NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure
Source
Elizabeth Goodman
Author
Elizabeth Goodman
Date
Why it matters
Identical hardware does not mean identical throughput — the gap is large enough to matter when comparing providers.
Key quotes
“Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We routinely see 8% to 12% gaps between partner deployments and the corresponding NVIDIA reference architecture (RA) on the same workload, same model, same global batch size.”
NVIDIA
“The CPU on the partner cluster was configured with C-states limited to C1 in BIOS. This is a common “low-latency” default that is actively wrong for AI training workloads.”
NVIDIA
“Allowing the idle cores to drop to C6 freed power headroom, enabling the busy cores to climb to 3.8 GHz and recover roughly 4% on this workload.”
NVIDIA
“Don’t increase QPS everywhere. QPS is fabric- and workload-dependent.”
NVIDIA
“If the path doesn’t resolve, NCCL fails silently with no error making this one of the harder gaps to diagnose without knowing where to look.”
articleBenchmarking LLM Inference at Scale with AIPerf
articleTensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
articleDense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
articleScaling Federated Learning Across Docker, Kubernetes, and Slurm with NVIDIA FLARE
articleH100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over TimeDylan Patel
blogNVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 DebutZhihan Jiang
postGB300 NVL72 claimed at 7x perf-per-dollar over H200 for agentic inferenceSemiAnalysis