AWS Machine Learning Blog (Amazon Web Services) · Managed Dynamo inference

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

AWS has added support for NVIDIA Dynamo, an open source distributed inference framework, to SageMaker HyperPod Inference. The add-on supports multi-node and disaggregated prefill and decode serving, KV-aware routing, and engines including vLLM, SGLang and TensorRT-LLM, with HyperPod handling node health and replacement. In the benchmark given, disaggregated serving of Qwen3-235B-A22B-FP8 on two p5en.48xlarge instances reached 2.05× the highest aggregated throughput (207 versus 101 output tokens per second per GPU) at the stated latency targets.

Open the source ↗

Similar Developer Tools releases

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.