AI inference gets a new tier as context windows grow

AI inference is experiencing a new tier as context windows grow larger, according to experts at Solidigm, Vast Data, and Supermicro. As agentic AI systems analyze, act and reassess, they generate longer contexts and more data that need to be accessed quickly during inference. This creates new storage and memory demands for AI workloads, beyond what graphics processing unit memory can handle.

The companies discussed how storage infrastructure can provide additional tiers to balance proximity, capacity, and speed for the data load. Solidigm supplies fast SSDs for cached data, Supermicro integrates them into rack-scale systems, and Vast Data's AI Operating System provides network storage and data services. The middle tier, enabled by Non-Volatile Memory Express SSDs, allows context data to live in places previously considered unsuitable.

KV cache, a significant optimization for AI inference, can replace compute with storage, reducing GPU expense and latency. Supermicro's Context Memory eXtension (CMX) targets large AI clusters with substantial data demands, while other deployments may require different combinations of memory, local SSDs, and network storage. Vast Data has tested KV cache offload, achieving 20 times faster time-to-first-token and 90% savings in GPU time.

Solidigm's SSDs address different points in the hierarchy, and Supermicro integrates them into customizable systems. The storage infrastructure's ability to match storage performance and capacity to the workload is crucial for AI applications.