Urgent.News

What's breaking now, across hundreds of outlets.

Tech

2026-11-22: PagedAttention explained from scratch

This post works through Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv:2309.06180). The paper starts from a waste problem: in the serving systems it measured, only 20.4% to 38.2% of the KV cache memory held real token states. It names three causes: reservation of room for the maximum length, internal fragmentation inside each reservation,…

This post breaks down Kwon et al.'s paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv:2309.06180). The study observes that in serving systems, only 20.4% to 38.2% of the KV cache memory is occupied by actual token states. Three causes for this waste are identified: allocating space for the maximum length, internal fragmentation within each allocation, and external fragmentation between allocations.

PagedAttention, introduced by the paper and implemented in the vLLM server, draws inspiration from operating system paging. It functions by reserving only small, fixed-size blocks of memory for each request, maintaining a table to track where these blocks are located, and allowing requests that share text to point to the same blocks. These blocks are copied only when modifications are required.

The paper reports a 2-4x increase in throughput compared to FasterTransformer and Orca at the same latency. To illustrate this, the author walks through a single request using the OPT-13B model (40 layers, 5120 hidden size, fp16 numbers). A prompt of 100 tokens generating 50 tokens results in 150 tokens. The reserved memory for this request is 1.56 GB (1.677 TB), whereas the actual usage is only 117.2 MiB (92.7% of the reserved memory).

Two types of waste are discussed: unused slots and internal fragmentation. The unused slots represent wasted memory, while internal fragmentation occurs due to the over-provisioning of memory for the maximum length. External fragmentation arises from the presence of free memory in small, non-contiguous blocks. The paper emphasizes these two sources of waste as distinct, despite their interconnectedness.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

Running Docker Sandboxes on Raspberry Pi 5

Docker Sandboxes (sbx v0.54.0) runs on an 8 GB Raspberry Pi 5: local microVMs start, run Docker inside themselves and enforce a network policy.

  • DockerSandboxes available on 8 GB Raspberry Pi 5
  • Installation involves downloading .deb package and granting KVM access
  • Network policy enforced, ports and files managed within sandbox

How to Pick a Gamertag That Fits Every Platform

Your gamertag is the first thing other players see. It shows up in lobbies, kill feeds, friend lists, and scoreboards — often in a font size you cannot control.

  • Choose a simple, readable gamertag for easy identification in matches.
  • Avoid leetspeak, personal info, impersonation, trademarked names, and complex capitalization.

Sending a Webhook Is Easy. Delivering It Reliably Is the Hard Part.

A webhook often starts out as a small feature. After an order is paid or a booking is confirmed, your backend makes an HTTP POST to another service. Done.

  • Webhooks appear simple but receiving service is complex
  • Challenges include timeouts, duplicates, failed acknowledgements
  • Hookstride addresses these with retries, signatures, logs

More from Sunday 22 November →