NVIDIA · Training Recovery Service

NVIDIA Recovery

NVIDIA Recovery is an open-source, workload-level service that detects interruptions in distributed training jobs from application logs, GPU metrics and scheduler signals, attributes failures, isolates implicated nodes and restarts workloads from checkpoints according to configured policy. It integrates with Slurm through a reference Advisor Controller deployable via Helm chart, and new deployments start in monitor mode so operators can check signal quality before enabling autonomous actions. In a production study on one training cluster from 9 July to 9 October, it covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.