Keep Distributed Training Running with Automated Workload Recovery

Shows a concrete design for auto-recovering failed distributed training jobs: monitor mode first, then autonomous actions, with an audit trail. Reported production latencies give a baseline for what automated recovery achieves at scale.

• NVIDIA positions Recovery above in-job libraries such as NVRx: NVRx handles hangs and restarts inside one job, while Recovery manages restarts, placement and node exclusion across all the workloads sharing a cluster.
• The reported 94-second mean recovery-action latency ends at the automated action, NVIDIA notes. Scheduling and checkpoint restoration still decide when productive training actually resumes.
• A watchdog lets normal scheduling proceed if a recovery decision takes too long or fails, and retries can be stopped when a job fails to save a new checkpoint across a configurable number of attempts.
• For Slurm, an SSH mode lets the control plane drive the scheduler remotely without installing an in-cluster service, which NVIDIA offers for sites where a new component needs a change window or security review.

• articleDiagnose Distributed AI Workload Failures with NVIDIA Attribution
• articleMemory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4
• articleHow NVIDIA OpenShell Reference Architecture Secures and Governs AI Agent Workloads at Scale
• articleCode, Test, and Deploy on One Machine with NVIDIA RTX Spark

• articleValidate GPU Cluster Readiness Before AI Workloads LandMichelle Horton
• articleHow NVIDIA NVLink 6 Delivers Multi-Layer Resiliency for AI FactoriesElizabeth Goodman
• articleDiagnose Distributed AI Workload Failures with NVIDIA AttributionElizabeth Goodman

Checking sign-in…

Loading comments…