Why My Prefix Cache Missed Identical Requests

In a test with my Rust LLM inference engine mini-vllm-rs on a Mac mini GPU, I enabled prefix caching and submitted identical requests to a cold cache. Neither request was able to reuse any tokens since the cache only published reusable pages after a request finished. Later requests reused correctly published prefixes, but to enable concurrent reuse I had to make new decisions about when cached pages should become visible, whether other requests should wait for them, and when those waits should end.

Testing a 4-bit quantized Qwen2.5 0.5B model, I benchmarked two warm-cache strategies against a cold-cache problem. With incremental publication, after each batched model forward the engine published newly completed pages containing committed tokens. This let subsequent requests restore 480 of their 493 prompt tokens, reducing the time to first token by 48-93% compared to completion-only publication.

However, incremental publication did not eliminate duplicate work for simultaneous cold arrivals. When a request started execution, it could still only find part of the prompt or nothing yet, and compute the rest itself. To coordinate prompt computation, I added exact-input single flight. One request became the leader, while other identical requests waited as followers. This improved mean TTFT and slowest request completion latency compared to independent execution, but introduced a new waiting cost.

To resolve this, I added a second release policy where once the leader finished prefill, the engine published its completed prefix pages and made followers eligible for admission. This reduced mean TTFT by 56.9% for 2 concurrent requests, 64.9% for 8, compared to independent execution. The early-release policy, publishing the prompt prefix when prefill finishes, reduced mean TTFT by 72% for 2 concurrent requests but increased the slowest request's completion latency by 14%.