NVIDIA · Failure Attribution

NVIDIA Attribution

NVIDIA Attribution is an open-source, workload-centric failure diagnosis system that correlates logs, metrics, telemetry and scheduler state into a structured diagnosis with root cause, affected nodes, supporting symptoms and an evidence trail. It ships as an Attribution Service with a REST API and a Python Blueprint that connects Slurm, Kueue, Loki, Kubernetes pod logs and PromQL metrics. The post says it runs in production across NVIDIA AI infrastructure, analysing millions of workloads weekly across hundreds of thousands of GPUs, and uses deterministic rules so results are reproducible.