Amazon Web Services (AWS) · KV cache routing

SageMaker AI KV cache aware routing

Amazon SageMaker AI now offers KV cache-aware routing (KV_AWARE) for real-time endpoints. The model container reports its cached contents, and SageMaker AI keeps a managed index that sends each request to the instance holding the longest cached match for its prompt, unless that instance is too busy. It needs at least two instances, one environment variable on the model (SM_VLLM_KV_EVENTS_CONFIG) and the RoutingStrategy setting in the endpoint configuration. Requests fall back to load-aware routing when no useful match exists, and the page gives added latency of a few milliseconds for short prompts.

Open the source ↗

Similar Developer Tools releases

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.