Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training

Onur Satici, a Staff Engineer at SpiralDB and a core maintainer of Vortex, an open-source columnar file format now part of the Linux Foundation, presents his work on high-throughput data loading. He explains how Vortex revolutionizes the process, eliminating bottlenecks and streaming data directly to GPUs at speeds up to 60 Gbps without the need for upfront data reprocessing. Vortex achieves this by utilizing cascading lightweight encodings, layout-based segment pruning, and zero-copy memory pipelines.

He emphasizes the importance of GPU acceleration and optimizing data loading for ML training. Satici demonstrates how Vortex can process a 4K video, with data flowing from S3, through the network card, over the CPU, and to the GPU at around 13 gigabits per second. He showcases a visualization of this data flow and explains how selecting only one column in the projection expression keeps the same bandwidth and maintains the video quality.

Satici discusses the two main taxes associated with GPU utilization: movement tax and decision tax. Movement tax refers to the time and resources spent moving data from the source to the GPU, primarily due to CPU decompression and NVMe throughput limitations. Decision tax arises when iterating on the training workload, requiring reprocessing of the entire dataset, which is time-consuming and inefficient.

Satici concludes by highlighting the potential of Vortex to streamline high-throughput data loading, making GPUs more accessible and efficient for ML training.