Building with AI

Catching silent GPU errors in large-scale LLM training on SageMaker

Amazon details a method to detect silent data corruption in SageMaker HyperPod by adding three checks to PyTorch training loops, preventing wasted compute on faulty hardware.

Illustration of a GPU chip with a subtle defect affecting data flow, protected by a security shield.
Illustration generated for this article

Amazon Web Services has published a technical guide on detecting silent data corruption (SDC) during large language model training on SageMaker HyperPod. The approach integrates three specific validation checks into PyTorch DistributedDataParallel loops to identify hardware faults that standard monitoring tools miss. This method allows engineers to isolate corrupted nodes and recover training jobs before significant compute resources are wasted.

What happened

Silent data corruption occurs when hardware components, such as GPUs, return incorrect but plausible numerical results without triggering any error flags. In distributed training environments, these erroneous values propagate through gradients and weights across all model replicas. Because the loss metrics may appear normal for hours, teams often only discover the issue after noticing unexplained model divergence or poor performance, by which time substantial compute costs have already been incurred.

The scale of this problem is significant. During a 54-day pre-training period for Llama 3 405B on up to 16,000 H100 GPUs, Meta attributed 6 out of 419 unexpected job interruptions to silent data corruption. These were only the cases that caused immediate job failures and were successfully traced back to SDC. Standard SageMaker HyperPod features, such as automatic node replacement and deep health checks, fail to catch SDC because the hardware does not report any faults.

To address this, AWS engineers developed a solution that adds detection logic directly into the training job. When a check identifies potential corruption, the system maps the suspect process rank to its physical node, contains the issue, and triggers SageMaker HyperPod to replace the faulty hardware. This shifts the responsibility of detection from infrastructure monitoring to the training application itself.

How it works

The solution relies on three distinct checks within the training loop, each targeting different stages where corruption can occur. First, a non-finite gradient check monitors for NaN or Infinity values in local gradients before and after the all-reduce operation. This catches errors that cause numerical overflow in the forward or backward passes. Second, a ring replay audit recomputes micro-batches from other nodes on audit steps, comparing loss values bit-for-bit and gradient fingerprints to detect finite wrong values in the compute path. Third, a weight fingerprint comparison verifies that all replicas have identical weights after the optimizer step, catching corruption in stored weights or the update process.

These checks use efficient fingerprinting rather than transmitting full tensors. Each tensor is reduced to two 64-bit sums: one plain sum of raw bit patterns and one weighted by position. This method detects changes to individual elements while minimizing communication overhead. The position-weighted sum is crucial for catching swapped values that might leave the plain sum unchanged. If any check fails, the system identifies the specific GPU rank responsible.

Once a suspect rank is identified, the system leverages Amazon Elastic Kubernetes Service (EKS) integration to map the rank to a specific instance ID. The job is stopped, the node is cordoned, and GPU metrics are reviewed via SageMaker HyperPod observability dashboards. If confirmed, the node is replaced, and training resumes from a checkpoint saved prior to the suspected corruption window.

Key details

  • Silent data corruption stems from manufacturing defects or silicon aging in compute units, particularly multiply-accumulate lanes in tensor cores, which lack ECC protection.
  • Standard integrity checks like ECC, Xid errors, and DCGM health checks miss SDC because they guard memory, driver faults, or transport layers, not the compute result itself.
  • The solution requires at least three GPU nodes to enable majority voting for weight comparisons and ring replay audits.
  • Local gradient norm comparisons are ineffective for detecting finite SDC, as healthy variance across replicas can exceed fault signals by orders of magnitude.
  • The implementation is designed for PyTorch DistributedDataParallel and relies on deterministic kernels for accurate gradient fingerprinting.
  • Full code for the solution is available in an accompanying GitHub repository provided by AWS.

Why it matters

For engineers building large-scale AI models, silent data corruption represents a hidden cost that can derail months of training. Traditional infrastructure monitoring is blind to these errors because the hardware appears healthy while producing incorrect math. By moving detection into the training loop, teams can prevent the propagation of bad weights and avoid wasting thousands of GPU hours on compromised jobs. This is especially critical as model sizes grow and training runs extend over weeks or months.

The approach also highlights the limitations of relying solely on hardware-level diagnostics. As silicon ages under heavy thermal and voltage stress, marginal defects can emerge that only manifest under specific workload patterns. Application-level verification provides a necessary safety net, ensuring that the mathematical integrity of the training process is maintained regardless of underlying hardware quirks. This method enables more robust fault tolerance strategies for production ML systems.

What you can do

  • Implement non-finite gradient checks in your PyTorch DDP hooks to catch NaN or Inf values before they spread via all-reduce.
  • Add periodic ring replay audits to your training loop, recomputing micro-batches from peer nodes to verify gradient fingerprints.
  • Compare weight fingerprints across all replicas after optimizer steps to detect corruption in stored parameters.
  • Ensure your SageMaker HyperPod cluster is orchestrated with EKS and has the observability add-on enabled for node mapping and metric correlation.
  • Use seeded samplers in your data pipeline to allow deterministic reconstruction of micro-batches for audit purposes.
  • Store checkpoints on shared storage like Amazon FSx for Lustre or S3 to facilitate quick recovery after node replacement.
  • #aws
  • #sagemaker
  • #llm training
  • #gpu
  • #fault tolerance

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories