Lightbits set to release KV cache engine to boost GPU performance

Lightbits Labs has announced the release of Inferra, a software engine designed to enhance AI inference economics and performance by moving key-value cache data beyond graphics processing units' high-bandwidth memory. The company, known for inventing NVMe over TCP, positions Inferra for neocloud providers and enterprises handling large language models with extensive context windows or many concurrent sessions.

Inferra manages KV cache across GPU HBM, DRAM, and NVMe storage, employing predictive prefetching to ensure data near the GPU before it's required. KV caches store intermediate attention data during model processing, with their size growing with conversation or document length and consuming valuable GPU memory. When needed cache data is unavailable, retrieval from slower memory or recomputation occurs, idle resources, and increased response times.

Inferring which data the GPU will need, Chettuvetty claims Inferra can reduce time to first token by over 100 times in certain long-context workloads, support context windows of over 10 million tokens on regular hardware, and increase concurrent session density by more than 16 times. These claims are based on company benchmarks and extrapolations, unverified independently.

With nearly 99.9% cache hit rates in most tests, Inferra also offers quality-of-service controls, encryption, isolation between tenants, and data relocation when sessions migrate to another GPU cluster. Comparing the approach to CPU techniques developed as CPU speeds outpaced memory performance, Arthur Rasmusson, AI architecture director at Lightbits, highlights the algorithms' differences in dealing with LLM inference.

Inferring capacity and retrieval speed, the engine breaks context into smaller blocks, feeding the GPU just-in-time, allowing operators to maintain larger caches and accommodate more users on the same infrastructure. Benefits are expected most with retrieval-augmented generation, AI agents, lengthy prompts, and heavily shared GPU services; however, the value may be limited for organizations with oversized private GPU clusters, few users, and contexts already fitting in memory.

By dispersing cache from GPU, Inferra introduces latency, which is mitigated by predicting demand far enough in advance to stage data before the GPU requests it. Faster networks and storage can reduce the required prediction window and limit forecast inaccuracies. Lightbits is currently running production pilots and targeting neocloud providers, whose need to increase utilization and margins makes them quicker adopters than hyperscalers. The company will showcase Inferra at the AI Infra Summit in Santa Clara next week.