Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

vLLM is an open-source tool for efficiently running large language models (LLMs). The inference process in vLLM can be divided into two main stages: prefill and decode. Prefill involves generating a key-value (KV) cache based on the input prompt, while decode utilizes this cache to generate the model's output token by token. These two stages have different computational, memory, and scalability requirements.

To optimize resource utilization, throughput, and latency, prefill and decode can be separated into different engine instances, a technique known as prefill-decode (P-D) disaggregation. However, this approach requires efficient transfer of the KV cache from the prefill instance to the decode instance before token generation can begin, making the KV transfer process critical to the time-to-first-token (TTFT).

One method vLLM uses for this transfer is the NVIDIA Inference Xfer Library (NIXL) connector, which utilizes RDMA-supported interconnects like AWS Elastic Fabric Adapter (AWS EFA) to transfer KV blocks directly between GPU or AWS Trainium device memory without staging through host memory. By default, NIXL operates in a pull-based mode, where the decode instance waits for the prefill instance to complete prefilling, then allocates destination blocks and initiates the NIXL READ operation to pull the KV blocks.

This sequential process extends the TTFT critical path, as it involves coordination and setup steps that run after prefill and before decoding can begin. To address this issue, vLLM introduces a push-based NIXL connector that moves the transfer coordination and setup off the critical path, allowing the decode instance to register its target blocks in advance while the prefill instance continues computing the KV.

This architecture removes the round-trip delay associated with the pull-based mode and moves the transfer-setup time to the prefill side, where it runs on a background thread. The push-based connector coordinates three actors: the decode instance registers its target blocks ahead of time, the prefill instance matches each request and initiates the transfer, and a dedicated background thread carries the KV write off the engine's forward-pass loop.

This design improves TTFT and decode throughput by overlapping producer and consumer work, eliminating the need for a consumer-side ID-discovery round trip, and offloading transfer-setup time from the latency-critical decode instance.