Amazon Web Services (AWS) · KV cache routing
SageMaker AI KV cache aware routing
Amazon SageMaker AI now offers KV cache-aware routing (KV_AWARE) for real-time endpoints. The model container reports its cached contents, and SageMaker AI keeps a managed index that sends each request to the instance holding the longest cached match for its prompt, unless that instance is too busy. It needs at least two instances, one environment variable on the model (SM_VLLM_KV_EVENTS_CONFIG) and the RoutingStrategy setting in the endpoint configuration. Requests fall back to load-aware routing when no useful match exists, and the page gives added latency of a few milliseconds for short prompts.
Open the source ↗Similar Developer Tools releases
- Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
AWS Machine Learning Blog (Amazon Web Services) · Managed Dynamo inference · Week 48, 2026
- inference-optimisation
sybil-solutions · Inference Optimisation Skill · Week 48, 2026
- Together Compiler
Together AI · Agent-driven GPU compiler · Week 48, 2026
- cuBLAS Ozaki-II and 2M FP64 Algorithms
NVIDIA · FP64 GEMM Emulation · Week 47, 2026
- TPU Sync
Google · KV Cache Transfer · Week 47, 2026
- SGLang-Neuron
RadixArk with Annapurna Labs (AWS) · Trainium inference plugin · Week 47, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.