NVIDIA Attribution: Open-source tool for diagnosing distributed AI failures

NVIDIA has released NVIDIA Attribution as an open-source project to help engineers diagnose failures in distributed AI workloads. Published on November 2, 2026, this tool addresses the complexity of debugging large-scale GPU clusters by correlating disparate data sources into a single, structured diagnosis. It moves beyond simple error reporting to explain why a workload failed.

What happened

Distributed training jobs often fail due to cascading errors where a single hardware or network issue triggers thousands of downstream symptoms. A fault in one GPU might manifest as communication timeouts across many other ranks, making it difficult to distinguish the root cause from the resulting noise. Traditional monitoring tools typically report where the workload stopped, not where the failure originated, leading to prolonged investigation times.

To solve this, NVIDIA open-sourced its internal workload-centric failure-attribution system. The tool, which has been used in production across NVIDIA AI infrastructure for years, analyzes millions of workloads weekly across hundreds of thousands of GPUs. By correlating workload-scoped logs, metrics, telemetry, and scheduler state, it produces a reproducible diagnosis that includes the outcome, root cause, affected nodes, and an evidence trail.

The release includes two main deployable components: the Attribution Service and the Attribution Blueprint. The Service exposes the core attribution engine via a REST API, while the Blueprint acts as a Python reference orchestrator. The Blueprint connects cluster schedulers and telemetry sources to the Service, allowing teams to integrate the tool into their existing infrastructure without rebuilding the core logic.

How it works

The attribution process follows four deterministic stages to ensure reproducibility. First, the Blueprint scopes the workload by identifying the specific attempt using scheduler identifiers or execution windows. Second, it collects observations from various connectors, including application logs, syslogs, GPU driver messages, and network metrics. These inputs are normalized into structured observation records.

Next, the Service extracts and correlates events using a rules engine. It identifies significant events such as GPU XIDs, NCCL failures, and scheduler transitions. Deterministic rules evaluate the timing and interaction of these events to separate downstream symptoms from the primary cause. For example, if a GPU reports a signal integrity issue and other ranks report communication errors, the system identifies the hardware fault as the root cause and the communication errors as supporting symptoms.

Finally, the system returns a structured result containing the terminal state, attributed domain, and affected nodes. The output distinguishes between definitive root causes and ambiguous situations where evidence is incomplete. This structured format allows downstream systems, such as incident management or automated recovery tools, to consume the diagnosis without parsing raw, product-specific logs.

Key details

• The system operates through two components: the Attribution Service for analysis and the Attribution Blueprint for orchestration and data collection.
• It supports execution-context connectors for Slurm, Kueue, and static inputs, along with log and metric connectors for Loki, local files, Kubernetes pod logs, and PromQL.
• In production, the system has identified fabric failure patterns around specific host and switch ports by correlating port errors and connection resets.
• The attribution engine uses deterministic rules to evaluate event timing, ensuring that results are reproducible and customizable for different environments.
• The output includes a quality indicator that reflects uncertainty when multiple causes fit the evidence, avoiding false confidence in incomplete diagnoses.
• Developers can extend the system by adding new rules for failure modes or implementing connectors for custom schedulers and telemetry backends.

Why it matters

For software engineers and ML infrastructure leads, reducing mean time to resolution (MTTR) is critical for maintaining high utilization of expensive GPU resources. Manual debugging of distributed failures is time-consuming and error-prone, often requiring experts to sift through gigabytes of logs. NVIDIA Attribution automates this correlation, allowing teams to focus on fixing infrastructure issues rather than hunting for them. By providing a clear evidence trail, it also facilitates better communication between software teams and hardware operations.

The open-source nature of the project allows organizations to adapt the tool to their specific stack. Since the core logic is separated from the integration layer, teams can apply the same attribution rules across different clusters while customizing how data is collected and stored. This flexibility supports diverse environments, from small research clusters to massive production fleets, enabling consistent failure analysis practices across an organization.

What you can do

• Clone the NVIDIA Attribution repository from GitHub to access the source code and sample workloads.
• Follow the quickstart guide to run the sample workflow using Docker and the uv package manager.
• Inspect the generated diagnoses in the local dashboard to understand how the system correlates events and identifies root causes.
• Review the Blueprint configuration reference to connect your own scheduler and telemetry backends, such as Slurm or Kubernetes.
• Examine the included declarative and Python rules to learn how to encode new failure patterns for your specific hardware or software stack.
• Integrate the Attribution Blueprint Result into your existing incident management or fleet-recovery systems to automate responses to common failures.