vLLM push-based KV transfer cuts latency in disaggregated inference

Amazon Web Services has introduced a push-based key-value (KV) transfer mechanism for the open-source vLLM inference engine. This update targets disaggregated prefill-decode architectures, aiming to significantly reduce time-to-first-token (TTFT) and improve throughput during token generation. The feature was validated on Amazon EC2 P5en instances using the Qwen3-32B model.

What happened

Large language model inference consists of two distinct phases: prefill, which processes the input prompt to build the KV cache, and decode, which generates output tokens using that cache. In disaggregated serving, these phases run on separate engine instances to allow independent scaling and optimization. However, this separation requires transferring the KV cache from the prefill instance to the decode instance before generation can begin, making the efficiency of this transfer critical for overall latency.

Previously, vLLM used a pull-based method via the NVIDIA Inference Xfer Library (NIXL) connector. In this default mode, the decode instance waits for the prefill to finish, receives metadata from a router, allocates memory, and then pulls the data. These steps happen sequentially after prefill completes, adding significant delay to the critical path. The new push-based connector changes this flow by overlapping coordination tasks with the prefill computation itself.

AWS engineers developed and tested this feature on clusters connected with AWS Elastic Fabric Adapter (EFA), which supports Remote Direct Memory Access (RDMA). While initially built for AWS Trainium instances, the contribution is accelerator-agnostic and works with GPUs as well. Benchmarks show that under high load, the push mode can reduce TTFT by up to three times compared to the pull mode, while also improving time-per-output-token (TPOT) by roughly 30 percent.

How it works

The push-based architecture removes serialization bottlenecks by allowing the decode instance to register its target memory blocks in advance. Instead of waiting for prefill to finish before preparing for data receipt, the decode side sends a registration notification to the prefill instance as soon as it allocates local blocks. This means the destination is known before the source data is even ready.

On the prefill side, a dedicated background thread handles the transfer preparation. Once the prefill computation finishes and the registration from the decoder arrives, the main engine loop hands off the KV blocks to this background thread. The thread establishes the NIXL connection, maps the block IDs, and issues an NIXL WRITE operation to copy data directly into the decoder’s registered memory. This keeps the heavy lifting of descriptor preparation and transfer submission off the latency-critical main loops of both instances.

The system uses a rendezvous protocol to handle timing differences. If the registration arrives before prefill finishes, the producer holds it. If prefill finishes first, it holds the blocks until the registration arrives. Whichever event happens second triggers the immediate transfer. The decoder counts completion notifications from each producer rank and only proceeds to token generation once all expected data has arrived.

Key details

• Performance gain: Up to 3x reduction in TTFT and 30% improvement in TPOT for Qwen3-32B on 1P1D deployments at high request rates.
• Hardware tested: Amazon EC2 P5en instances with EFA, using GPUDirect RDMA for direct GPU-to-GPU memory transfer.
• Mechanism: Uses NIXL WRITE operations initiated by the prefill instance, rather than NIXL READ initiated by the decode instance.
• Threading: A background thread on the prefill worker handles transfer setup, preventing stalls in the main model execution loop.
• Trade-off: Push mode requires selecting the decode instance earlier in the process, which may reduce flexibility for load-aware routing compared to pull mode.
• Configuration: Users enable this by setting both prefill and decode servers to use NixlPushConnector instead of the default NixlConnector.

Why it matters

For engineers building high-throughput LLM services, disaggregated serving is a common strategy to manage resource usage efficiently. However, the overhead of moving large KV caches between nodes can negate the benefits if not handled carefully. The traditional pull model introduces a synchronous wait period where the decode instance is idle, waiting for metadata and then pulling data. This delay becomes pronounced as context lengths grow and request rates increase, directly impacting user-perceived latency.

By shifting the coordination work to a background thread and overlapping it with prefill computation, the push model ensures that the decode instance is ready to receive data the moment it is available. This reduces the idle time on the critical path, leading to faster response times for users. It also prevents the decode worker from being bogged down by transfer setup tasks, allowing it to maintain consistent token generation speeds even under heavy concurrency.

This development highlights the importance of low-level infrastructure optimizations in AI serving. As models grow larger and contexts longer, the efficiency of data movement between compute nodes becomes just as important as the speed of the matrix multiplications themselves. Engineers deploying vLLM on AWS or similar RDMA-enabled infrastructure should consider this option to maximize performance.

What you can do

• Evaluate your current vLLM deployment to see if disaggregated prefill-decode serving is appropriate for your workload.
• Test the NixlPushConnector in a staging environment with realistic traffic patterns to measure TTFT improvements.
• Ensure your infrastructure supports RDMA, such as AWS EFA, to leverage direct memory access features.
• Monitor the trade-off between early decode instance selection and dynamic load balancing needs in your router configuration.
• Disable prefix caching during benchmarking to isolate the transfer performance, as caching can mask transfer costs.
• Review the vLLM documentation for specific configuration flags required to switch from pull to push mode.