Amazon Web Services · Global Inference Routing

Amazon SageMaker HyperPod Global Inference Gateway

AWS has launched a SageMaker-managed gateway that gives a single endpoint across every HyperPod inference cluster a customer registers, in one Region or several. It scores each healthy cluster on KV cache headroom, queue depth, Region proximity and error rate, routes each request to the best one over AWS PrivateLink, and fails over automatically when a cluster goes down. Routing weights are configurable through the CreateGlobalInferenceGateway and UpdateGlobalInferenceGateway APIs, and requests are authenticated with SigV4 and IAM.

Open the source ↗

Similar Developer Tools releases

More from Amazon Web Services

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.