What happens when a GPU reads memory

When a GPU's graphics processing unit (GPU) accesses memory, it follows a specific path through the hardware. In this case, we are examining an RTX 4090. Our focus is on a critical SASS (Stream Processor Assembly Stream) instruction called global load, which retrieves data from global memory into a register for further processing. This process involves several stages, including address resolution, load/store unit operations, and cache access.

Starting with the LDG.E R4, [R4.64] instruction, the GPU first retrieves the 64-bit address stored in registers R4 and R5. This address is then used to fetch the required data from global memory. The warp, consisting of multiple threads, reads the addresses from the register file in parallel. Each lane within the warp receives a 64-bit address, resulting in a total of 256 bytes being read from global memory.

After the addresses are resolved, the instruction dispatches to the load/store unit (LSU). Here, the LSU performs any necessary address arithmetic and sends the opcode, active lane mask, computed addresses, and output register number to the coalescer. The coalescer's role is to determine the minimum number of L1 cache sectors required to satisfy the request. In this scenario, the coalescer identifies that it needs to retrieve 4 contiguous 32-byte sectors, which correspond to a single cache line.

The coalescer sends a request to the L1 cache, which is organized into 128-byte lines. In this case, the requested sectors fit within a single cache line. The L1 cache's unit of organization is a set-associative structure, with each set containing four slots. The virtual address used in the program determines which set the requested line belongs to. A hash scheme is used to generate the set index, ensuring that power-of-2 strided accesses don't cause cache line churn.

If the requested data is found in the L1 cache, the load operation is complete. However, if the data is not found (a cache miss), the request must descend further into the memory hierarchy. An L1 cache miss incurs a latency of approximately 15.4 nanoseconds (40 cycles). Virtual memory introduces an additional layer of indirection between the program's addresses and the actual memory locations, requiring a translation process to locate the data in physical memory.

In summary, the GPU's journey to access memory involves resolving addresses, dispatching instructions, performing coalescing, accessing the L1 cache, and potentially traversing additional memory hierarchy levels if a cache miss occurs.