HPE puts storage to work on AI’s expensive memory problem

HPE is expanding the role of its Alletra Storage MP X10000 storage system by using it to manage inference state that would otherwise be discarded from expensive GPU memory. In tests validated by Kamiwaza using NVIDIA H200 GPUs and the Kamiwaza AI platform, the storage offload solution resulted in up to 20 times faster time to first token and 17 times higher effective inference throughput compared to running without the offload.

The GPUs did not become more efficient; they simply had less redundant work to perform. This innovation brings storage considerations into discussions about AI economics, as enterprises move from individual question answering to agents that perform multiple tasks based on a single request. Agentic workflows repeatedly process documents, system instructions, and conversation histories, with some repetition being necessary but unnecessary recalculations potentially avoided through cache reuse.

HPE's Alletra MP X10000 provides a shared cache tier for offloaded data, while RDMA for S3 accelerates the storage data path. By retrieving reusable context instead of recalculating it, the system can reduce the work required before a model starts generating its next answer. The Kamiwaza-validated tests used NVIDIA Nemotron 70B models and over 30 concurrent agents under large-context, multi-agent inference, demonstrating the potential benefits in real-world workloads.

However, the performance gains are workload-specific and depend on factors such as context length, concurrency, cache hit rates, and transfer overhead. Buyers should evaluate these factors alongside the percentage improvement to determine the true value of the solution. While cache offload is gaining traction in the market, HPE differentiates itself by integrating data preparation and governance with the storage solution, rather than simply offering another feature checkbox.

The ultimate goal is to ensure that applications can efficiently find relevant, authorized information and that the inference engine can effectively reuse computation. Ultimately, the focus should be on observable outcomes, such as cache hit rates, time to first token, sustained throughput, and the quality of retrieved information, rather than solely on benchmark results.