2026-11-22: PagedAttention explained from scratch
This post works through Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv:2309.06180). The paper starts from a waste problem: in the serving systems it measured, only 20.4% to 38.2% of the KV cache memory held real token states. It names three causes: reservation of room for the maximum length, internal fragmentation inside each reservation,…
This post breaks down Kwon et al.'s paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv:2309.06180). The study observes that in serving systems, only 20.4% to 38.2% of the KV cache memory is occupied by actual token states. Three causes for this waste are identified: allocating space for the maximum length, internal fragmentation within each allocation, and external fragmentation between allocations.
PagedAttention, introduced by the paper and implemented in the vLLM server, draws inspiration from operating system paging. It functions by reserving only small, fixed-size blocks of memory for each request, maintaining a table to track where these blocks are located, and allowing requests that share text to point to the same blocks. These blocks are copied only when modifications are required.
The paper reports a 2-4x increase in throughput compared to FasterTransformer and Orca at the same latency. To illustrate this, the author walks through a single request using the OPT-13B model (40 layers, 5120 hidden size, fp16 numbers). A prompt of 100 tokens generating 50 tokens results in 150 tokens. The reserved memory for this request is 1.56 GB (1.677 TB), whereas the actual usage is only 117.2 MiB (92.7% of the reserved memory).
Two types of waste are discussed: unused slots and internal fragmentation. The unused slots represent wasted memory, while internal fragmentation occurs due to the over-provisioning of memory for the maximum length. External fragmentation arises from the presence of free memory in small, non-contiguous blocks. The paper emphasizes these two sources of waste as distinct, despite their interconnectedness.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to