Diagnose Distributed AI Workload Failures with NVIDIA Attribution
Source
developer.nvidia.com
Author
Elizabeth Goodman
Date
Why it matters
Offers an open-source way to turn scattered logs, metrics and scheduler state into a reproducible root-cause diagnosis for failed distributed AI jobs. It has been used on fabric failures at large GPU scale.
Key takeaways · AI-distilled
NVIDIA's premise: the exception a distributed job reports shows where it stopped, not where the failure began, because one GPU or fabric event can surface as NCCL errors on other ranks and thousands of downstream messages.
In the post's worked example, an XID 149 NVLink signal-integrity fault on one node is named the root cause. The NCCL errors on other ranks are demoted to supporting symptoms but kept for inspection, so the job is reported as a hardware failure.
When more than one cause fits the evidence, the diagnosis marks sub_domain as ambiguous instead of forcing a verdict. NVIDIA says attribution quality depends on how complete the underlying telemetry is.
Downstream recovery and incident systems read the structured terminal_state and sub_domain fields instead of parsing product-specific logs, and several Blueprints can share one Attribution Service or each run their own.
articleKeep Distributed Training Running with Automated Workload Recovery
articleMemory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4
articleHow NVIDIA OpenShell Reference Architecture Secures and Governs AI Agent Workloads at Scale
articleCode, Test, and Deploy on One Machine with NVIDIA RTX Spark
