HPE puts storage to work on AI’s expensive memory problem

HPE is expanding the capabilities of its Alletra Storage MP X10000 by incorporating it into AI systems to manage inference state. This offloads expensive GPU memory, which can otherwise force the system to recalculate data later. In tests conducted with NVIDIA H200 GPUs, the Kamiwaza AI platform, and HPE's Alletra MP X10000, researchers observed a 20-fold increase in time to first token and a 17-fold boost in effective inference throughput compared to systems without KV cache offload.

The GPUs did not become more efficient; instead, they had less redundant work to perform. This development brings storage considerations into the conversation surrounding AI economics, particularly as enterprises scale from individual question answering to more complex, agent-based workflows. Agentic workflows involved in these tasks repeatedly process documents, system instructions, and conversation histories, leading to a significant amount of repeated computation.

KV cache can retain intermediate attention states, allowing compatible requests to reuse earlier computations and reduce the workload on the GPU. However, GPU memory is limited and pricier than one might expect. As long contexts and concurrent sessions vie for this memory, cache eviction can force the GPU to redo work, leading to reduced performance.

HPE's Alletra MP X10000 offers a shared tier for offloaded cache, supplemented with RDMA for S3 to accelerate the storage data path. This setup allows for the retrieval of reusable context, avoiding the need for the GPU to recalculate it. The principle of computing once and reusing often is not new, but the economic impact becomes more significant when considering the costs associated with repeated GPU work.

While the headline figure of 20 times faster is impressive, it is contingent on specific workload conditions such as context length, concurrency, cache hit rates, and transfer overhead. Buyers should evaluate these conditions alongside the performance improvements to determine the practical benefits. HPE's approach pairs the X10000 with Data Fabric for governed access to distributed data, aiming to improve retrieval precision and reduce irrelevant material in prompts.

This strategy can lower the cost of making a GPU read the same information multiple times. However, the effectiveness of this approach depends on the quality of the retrieval system and whether the system can retrieve relevant, authorized context efficiently. The broader economic implication of this technology lies in its ability to streamline data preparation, governance, and cache reuse, creating a more integrated and efficient AI system.

Organizations should assess the smooth operation of these components as a single system to gauge the real-world value of AI deployments.