Could HBF be the solution to the Memory bottleneck?

Artificial intelligence workloads are driving up demand for memory capacity and bandwidth, presenting an opportunity for a new storage technology known as high-bandwidth flash (HBF). Bernstein's research identifies HBF as a potential solution to infrastructure constraints by offering read bandwidth similar to high-bandwidth memory (HBM) but with eight to 16 times greater capacity at a lower cost per gigabyte.

This technology could enhance AI inference by decreasing the demand for expensive GPU memory and enabling systems to recall previously processed information instead of redundant computations.

As AI models grow larger and more complex, the memory required to process them surpasses the limits of high-bandwidth memory (HBM) and conventional DRAM, NAND flash, and hard disk drives. HBF could alleviate this issue through a tiered memory architecture, where HBM manages frequently accessed and latency-sensitive data, while HBF stores larger model parameters, less frequently used components, and reusable key-value (KV) cache data.

This could enable AI infrastructure providers to run larger models with fewer GPUs, reducing computing, power, and capital expenditure.

The adoption of HBF could create significant opportunities for NAND manufacturers, although at a higher manufacturing cost compared to conventional enterprise SSDs. Bernstein estimates that HBF manufacturing costs could be roughly 10 times greater than those of conventional enterprise SSDs, depending on the configuration. However, if priced at levels comparable to HBM while offering significantly greater capacity, HBF could generate attractive gross margins, with Bernstein estimating 86.5% for an eight-high configuration and 88.9% for a 16-high configuration.

Despite the potential, HBF faces several hurdles to widespread adoption. It is not a direct replacement for HBM and has limitations in latency, write bandwidth, endurance, and data access. HBF would necessitate changes to processor designs, system software, and careful optimization of data placement and memory management. Furthermore, the lack of official support from Nvidia, which has promoted alternative technologies like Storage Next and its CMX content memory storage platform, poses a challenge.

While companies like SanDisk, Seagate Technology, Western Digital, Samsung Electronics, SK hynix, and Micron are involved in the HBF ecosystem, broader industry adoption remains uncertain.