Bi-directional KV transfer cuts latency in multi-turn agentic workloads

Amazon Web Services has introduced a method to significantly lower latency in multi-turn agentic applications by enabling bi-directional key-value (KV) cache transfer within the vLLM serving framework. Published on October 19, 2026, this approach allows prefill and decode nodes to share cached context in both directions, eliminating redundant computation as conversation histories grow.

What happened

In typical multi-turn agentic workflows, an agent accumulates context by reasoning, invoking tools, and incorporating previous results. Clients usually resend the entire conversation history—including system instructions, prior prompts, tool outputs, and model responses—with each new turn. In Prefill-Decode (P-D) disaggregated serving architectures, requests are split between specialized nodes: a prefill node processes the prompt to build the KV cache, and a decode node retrieves these blocks to generate the response. However, standard implementations only transfer KV cache in one direction, from prefill to decode.

This one-way flow creates inefficiencies. After generating a response, the decode node holds KV entries for both the original prompt and the new tokens. When the next turn arrives, the prefill node cannot access these entries and must recompute KV projections for the previous response. Additionally, because decode nodes maintain session state while prefill nodes do not, active sessions on prefill nodes face higher risks of cache eviction. A full cache miss forces the prefill node to recompute the entire history, consuming accelerator capacity and increasing time to first token (TTFT).

To address this, AWS engineers extended the NVIDIA Inference Xfer Library (NIXL) KV connector in vLLM to support bi-directional transfer. This allows decode nodes to return previously computed KV blocks to prefill nodes for subsequent turns. The feature was validated on Amazon EC2 P5en instances connected via AWS Elastic Fabric Adapter (EFA), showing that it reduces TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 models in multi-turn benchmarks.

How it works

Bi-directional KV transfer relies on a KV-cache-aware router and enhanced block retention policies. The router acts as a lightweight, stateful control plane that maps client conversation IDs to decode node KV transfer parameters. It stores only block-location metadata, not the actual tensors, obtaining this information from the response stream. On the first turn, the cache misses, and the request follows the standard path. The router then saves the decode node’s metadata. On subsequent turns, the router retrieves this metadata, allowing the prefill node to reuse KV blocks from the decode node instead of recomputing them.

The decode node retains eligible KV blocks for a configurable time-to-live (TTL), defaulting to 480 seconds, rather than freeing them immediately after generation. This ensures the context remains available for the next turn. To prevent overhead from outweighing benefits for short prefixes, a kv_recompute_threshold setting controls when reuse occurs. If the number of reusable tokens is below this threshold (default 64), the prefill node recomputes locally; otherwise, it pulls KV from the decode node via a NIXL READ operation.

Key details

• Bi-directional KV transfer reduces TTFT by up to 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8 in tested configurations.
• The feature requires vLLM v0.21.0 or later with NixlConnector and must be enabled on both prefill and decode servers.
• AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA, allowing direct memory access between GPU or Trainium devices.
• Decode nodes retain KV blocks for a configurable TTL (default 480 seconds) to support reuse across turns.
• A kv_recompute_threshold (default 64 tokens) prevents unnecessary network overhead for short reusable prefixes.
• The KV-cache-aware router maintains in-memory mapping of conversation IDs to KV transfer parameters without storing tensors.

Why it matters

For engineers building conversational assistants and agentic applications, managing growing context windows is a primary bottleneck. As interaction histories lengthen, repeated prefill computation consumes significant accelerator resources and increases latency. By enabling prefill nodes to reuse KV blocks retained by decode nodes, bi-directional transfer mitigates this redundancy. This leads to more stable TTFT performance even as conversations extend, improving user experience and resource utilization.

The architecture also highlights the importance of efficient interconnects like AWS EFA for distributed inference. The small overhead of transferring KV blocks (30–130 ms in tests) is far outweighed by the savings in recomputation time, especially for larger models like GLM-5-FP8. This suggests that optimizing data movement between specialized nodes is as critical as optimizing model execution itself for scalable agentic systems.

What you can do

• Upgrade to vLLM v0.21.0 or later and enable bidirectional_kv_xfer on both prefill and decode servers.
• Implement a KV-cache-aware router to manage metadata flow between clients and serving nodes.
• Tune the decoder_kv_blocks_ttl parameter based on expected time between user turns to balance memory usage and reuse.
• Adjust the kv_recompute_threshold according to your model’s computation time and interconnect performance.
• Use AWS EFA-enabled instances like P5en for low-latency RDMA transfers if deploying on AWS infrastructure.
• Reproduce the benchmark using the provided guide to validate performance gains for your specific workload and model.