SageMaker KV-aware routing cuts LLM latency by tracking cache state
AWS introduces KV-aware routing for SageMaker AI, using real-time cache reports to reduce time-to-first-token by up to 97% in multi-turn LLM workloads.
Amazon Web Services has launched KV cache-aware routing for Amazon SageMaker AI real-time endpoints. This new feature, announced on November 17, 2026, allows large language model (LLM) serving infrastructure to route requests based on the actual content stored in GPU memory rather than static request prefixes. It targets developers building chat assistants and retrieval-augmented generation systems who need to minimize latency and maximize throughput under heavy load.
What happened
Previously, SageMaker offered prefix-aware routing, which hashed the beginning of a request payload to direct similar prompts to the same instance. While effective for stable system prompts, this method could not detect when an instance evicted cached data due to memory pressure. If an instance ran out of GPU memory and cleared its cache, the router would still send matching requests there, forcing the model to recompute tokens that were no longer available. This limitation also affected multi-turn conversations where the prompt length grew beyond the configured prefix hash, causing subsequent turns to land on different instances and lose context continuity.
KV-aware routing solves this by establishing a feedback loop between the model container and the SageMaker router. The container actively reports when it stores or evicts key-value (KV) cache entries. SageMaker maintains a managed index of these reports, mapping truncated hashes of cached content to specific instances. When a new request arrives, the endpoint tokenizes the prompt and queries this index to find which instance holds the longest matching prefix. The router then weighs the cache match length against the current load of candidate instances, sending the request to the best available option. If no instance has a useful cache match or all are busy, the system falls back to standard load-aware routing.
How it works
The mechanism relies on three components: the model container, a managed cache index, and the routing logic. To enable this, users must set the SM_VLLM_KV_EVENTS_CONFIG environment variable in their container, which triggers the publication of KV cache events. These events update the managed index with one-way, truncated hashes of the cached content. Crucially, the index stores only pointers to instances, never the actual prompts, token IDs, or tensor data, preserving privacy and minimizing overhead.
When a request hits the endpoint, the system tokenizes the input and checks the index. If Instance A holds the first 6,000 tokens of a 6,200-token conversation history, the router identifies it as the primary candidate. However, if Instance A is currently overloaded, the router skips it and selects the next-best instance with a substantial cache match. This dynamic balancing ensures that cache hits are prioritized without creating bottlenecks. The index automatically updates as the fleet scales, dropping entries for removed instances and starting fresh for new ones.
Key details
- Performance gains: In benchmarks using Llama 3.1 70B Instruct on
ml.p5.48xlargeinstances, KV-aware routing reduced p50 time-to-first-token by up to 97% and increased throughput by up to 234% compared to random routing. - Cache hit rates: The feature raised KV cache hit rates from a baseline of 3–48% to 80–93% in tested multi-turn chat and document Q&A scenarios.
- Container requirements: It requires the AWS vLLM Deep Learning Container with vLLM 0.33.0 or later, or Amazon Linux build
server-sagemaker-cuda-v2.8.0or later, with prefix caching enabled. - Routing fallback: If the cache lookup yields no matches or all matching instances are busy, the endpoint defaults to
LEAST_OUTSTANDING_REQUESTSrouting, ensuring requests are always served. - No manual tuning: Unlike prefix-aware routing, KV-aware routing does not require configuring a
PrefixLengthparameter, as it matches based on actual cached tokens. - Minimum scale: The strategy is active only when an endpoint has at least two instances or two copies of an inference component.
Why it matters
For engineering teams serving LLMs, the cost and latency of recomputing attention mechanisms for long contexts are significant bottlenecks. Multi-turn conversations and RAG applications often share large portions of text, such as system instructions or retrieved documents. By ensuring these shared prefixes remain cached on the instance handling the request, KV-aware routing drastically reduces the computational work required for each new token. This efficiency translates directly to lower infrastructure costs and faster response times for end users, particularly in high-concurrency environments where GPU memory is frequently contested.
This update also simplifies operational complexity. Developers no longer need to guess the optimal PrefixLength for their workload or worry about cache eviction invalidating their routing strategy. The system adapts automatically to the changing state of the GPU memory, providing a more robust solution for dynamic workloads. While prefix-aware routing remains a viable option for simpler, stable prompts, KV-aware routing offers a superior, self-correcting mechanism for complex, interactive AI applications.
What you can do
- Update your SageMaker model containers to use AWS vLLM Deep Learning Container v0.33.0 or later.
- Set the
SM_VLLM_KV_EVENTS_CONFIGenvironment variable in your container configuration to enable cache event reporting. - Modify your endpoint configuration to set
RoutingStrategytoKV_AWAREin theRoutingConfig. - Ensure your application sends requests in OpenAI-compatible Completions or Chat Completions format to support token-based matching.
- Monitor time-to-first-token and throughput metrics to verify cache hit improvements, as fallback events are not yet visible in the console.
- Test with at least two instances to activate the routing strategy, as it has no effect on single-instance deployments.



