Amazon Web Services · GPU Training Observability

Enhanced infrastructure observability in Amazon SageMaker HyperPod

AWS has added enhanced observability to Amazon SageMaker HyperPod, a feature for diagnosing GPU training problems on EKS and Slurm clusters. It combines a diagnostic summary, which uses CloudWatch's DetectGroupOutliers API to flag GPUs that deviate from their peers, with static rules for XID and ECC errors, EFA drops, throttling and overheating. Default metrics are collected every 30 seconds at roughly 1–2% CPU overhead and 250–300 MB memory, while advanced DCGM, Neuron and EFA metrics are opt-in. CloudWatch is now the default metric store, with Amazon Managed Service for Prometheus still available.

Open the source ↗

Similar Developer Tools releases

More from Amazon Web Services

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.