Amazon Web Services · Training Health Checks

Workload recovery and infrastructure checks on Amazon SageMaker HyperPod

AWS has added three capabilities to Amazon SageMaker HyperPod to catch training failures that node-level health checks miss. MultiNodeTrainingValidation is a deep health check that trains a small model on each instance and then across instances, failing nodes that produce NaN or infinite loss or that do not finish in time. The HyperPod Kubernetes Resource Monitoring Agent reports Kubernetes-layer problems on EKS clusters, and hung job detection flags stalled Slurm jobs and can cancel them so Slurm requeues them. The page does not state pricing or availability beyond the listed EKS and Slurm cluster types.

Open the source ↗

Similar Developer Tools releases

More from Amazon Web Services

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.