More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

Larger language models, longer contexts, and agentic applications are intensifying the demand for GPU memory, while supply constraints and rising costs are expected to persist in the near future. In response, NVIDIA collaborates closely with memory partners to swiftly adopt the latest memory technologies. Although NVIDIA initially embraced HBM3e and HBM4 technologies in its accelerated computing stack, the strategy extends beyond merely augmenting memory within the GPU.

Rather, it adopts a comprehensive codesign approach, integrating GPUs, CPUs, networking, storage, and software to extract maximal utility from memory at every layer of the AI infrastructure. This is significant, as an AI factory operates on a complex memory and storage hierarchy. The highest tier, GPU high-bandwidth memory (HBM), is the fastest, most expensive, and most competitive segment, with model weights and key-value (KV) cache vying for limited capacity.

Below, CPU memory, local SSDs, and shared storage are found, with each layer trading speed for increased capacity at a lower cost per bit. The efficiency of memory utilization at each level is crucial, as the amount of work achieved per byte depends on how well the entire hierarchy is utilized. By minimizing what must be stored, strategically placing it on the appropriate tier, and retrieving it back to the GPU promptly, NVIDIA demonstrates the most efficient memory usage across all layers.

This efficiency is evident in NVIDIA Blackwell Ultra and the Vera Rubin platform, where HBM capacity per GPU has increased by approximately 3%, while agentic throughput per dollar has improved by up to 30x on SemiAnalysis AgentX. Agentic inference imposes additional strain on HBM, as modern frontier models possess trillions of parameters, and KV cache, responsible for holding the context required for the model to comprehend each active request without reprocessing the same data, is also expanding.

Agentic trajectories, which carry prior context across multiple model and tool calls, can process up to 3 million tokens, thereby increasing the working memory required for each request. Retrieval-augmented generation (RAG) further compounds this by integrating relevant enterprise information into prompts, thereby generating additional context to process and retain.

During inference, HBM primarily accommodates both model parameters and KV cache, leading to heightened demand on HBM. While increasing HBM capacity per GPU appears as a straightforward solution to alleviate this pressure, it only provides an improvement up to the available bandwidth. Contemplating HBM as a warehouse and memory bandwidth as its loading docks, a larger warehouse offers additional inventory space, but the docks dictate how swiftly data can enter and exit.

When the docks reach capacity, augmenting the inventory takes longer, thereby delaying response times. If the delays surpass the required service level, the additional space remains underutilized, despite the heightened costs and power consumption. This explains why AI factories refrain from keeping all their inventory in the closest, fastest space.

Instead, what is in motion remains near the line, while what is waiting is moved to a larger, more affordable space nearby and returned to the GPU before it's needed. In essence, an AI factory confronts the same limitation as other operations. By incorporating more HBM, additional KV cache and concurrent users can be supported. However, as each user anticipates swift responses, eventually, memory bandwidth fails to supply data as swiftly as compute can utilize it, a phenomenon known as the memory wall.

Once surpassed, additional HBM capacity cannot be maximized, yet it still incurs costs and consumes power. The data must find a home, but not all of it necessitates the fastest and most expensive tier. This is the rationale behind managing data across a memory and storage hierarchy. Maximizing the value of every byte through full-stack codesign, NVIDIA enables every byte of HBM to contribute by alleviating pressure from model weights, KV cache, and RAG data.

Techniques employed include reducing model weights and distributing them across the NVL72 domain, employing the NVFP4 format to decrease the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform. Recent models, such as FLUX.2, Inkling, Beam 1.0, Meta Muse Spark, Step 5 Preview, and NVIDIA Nemotron 3.5 Super VL, have been released with NVFP4 checkpoints, showcasing the growing adoption of 4-bit precision for faster, more memory-efficient inference.

Within an NVL72 system, model weights can be distributed across up to 72 GPUs, enabling each GPU to hold a smaller portion of the model, thereby freeing up more HBM for KV cache and concurrent users. Further, reducing the KV-cache footprint and offloading inactive context to lower-cost storage tiers, NVFP4's application to the KV cache reduces its size compared to FP8, allowing more context to reside in HBM.

NVIDIA's Vera CPU aids in two ways: faster tool execution shortens the idle time of the KV cache during tool calls, and its substantial LPDDR memory, linked to NVIDIA Rubin GPUs through NVIDIA NVLink-C2C interconnect, offers a nearby location for paused context to await and return swiftly.