{
  "id": 15627129,
  "title": "Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM",
  "url": "https://urgent.news/2026/10/19/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-19T16:05:17.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm/"
  },
  "original_language": "en",
  "account": "vLLM is an open-source tool for efficiently running large language models (LLMs). The inference process in vLLM can be divided into two main stages: prefill and decode. Prefill involves generating a key-value (KV) cache based on the input prompt, while decode utilizes this cache to generate the model's output token by token. These two stages have different computational, memory, and scalability requirements. To optimize resource utilization, throughput, and latency, prefill and decode can be separated into different engine instances, a technique known as prefill-decode (P-D) disaggregation. However, this approach requires efficient transfer of the KV cache from the prefill instance to the decode instance before token generation can begin, making the KV transfer process critical to the time-to-first-token (TTFT). One method vLLM uses for this transfer is the NVIDIA Inference Xfer Library (NIXL) connector, which utilizes RDMA-supported interconnects like AWS Elastic Fabric Adapter (AWS EFA) to transfer KV blocks directly between GPU or AWS Trainium device memory without staging through host memory. By default, NIXL operates in a pull-based mode, where the decode instance waits for the prefill instance to complete prefilling, then allocates destination blocks and initiates the NIXL READ operation to pull the KV blocks. This sequential process extends the TTFT critical path, as it involves coordination and setup steps that run after prefill and before decoding can begin. To address this issue, vLLM introduces a push-based NIXL connector that moves the transfer coordination and setup off the critical path, allowing the decode instance to register its target blocks in advance while the prefill instance continues computing the KV. This architecture removes the round-trip delay associated with the pull-based mode and moves the transfer-setup time to the prefill side, where it runs on a background thread. The push-based connector coordinates three actors: the decode instance registers its target blocks ahead of time, the prefill instance matches each request and initiates the transfer, and a dedicated background thread carries the KV write off the engine's forward-pass loop. This design improves TTFT and decode throughput by overlapping producer and consumer work, eliminating the need for a consumer-side ID-discovery round trip, and offloading transfer-setup time from the latency-critical decode instance.",
  "summary": "vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "AWS Machine Learning",
        "title": "Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM",
        "url": "https://urgent.news/2026/10/19/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv",
        "published": "2026-10-19T16:03:23.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}