AWS Machine Learning Blog (Amazon Web Services) · Managed Dynamo inference
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
AWS has added support for NVIDIA Dynamo, an open source distributed inference framework, to SageMaker HyperPod Inference. The add-on supports multi-node and disaggregated prefill and decode serving, KV-aware routing, and engines including vLLM, SGLang and TensorRT-LLM, with HyperPod handling node health and replacement. In the benchmark given, disaggregated serving of Qwen3-235B-A22B-FP8 on two p5en.48xlarge instances reached 2.05× the highest aggregated throughput (207 versus 101 output tokens per second per GPU) at the stated latency targets.
Open the source ↗Similar Developer Tools releases
- inference-optimisation
sybil-solutions · Inference Optimisation Skill · Week 48, 2026
- SageMaker AI KV cache aware routing
Amazon Web Services (AWS) · KV cache routing · Week 48, 2026
- Together Compiler
Together AI · Agent-driven GPU compiler · Week 48, 2026
- cuBLAS Ozaki-II and 2M FP64 Algorithms
NVIDIA · FP64 GEMM Emulation · Week 47, 2026
- TPU Sync
Google · KV Cache Transfer · Week 47, 2026
- SGLang-Neuron
RadixArk with Annapurna Labs (AWS) · Trainium inference plugin · Week 47, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.