From API to GPU, Week 5: Tensors, the Data Structure Behind Every Model
Phase 2 of 8: Enough ML to understand inference. Week 5 of 32. Phase 1 was about running models. Phase 2 is about understanding what happens inside them, starting with the one data structure they are all built from: the tensor . This week I stop talking about model files and start touching the actual numbers, in PyTorch, on the GPU. If you write software, a tensor is a typed, multi-dimensional…
In the fifth phase of the eight-part series, the focus shifts from running models to comprehending the inner workings of machine learning models by exploring the fundamental data structure they are built upon: the tensor. In this week's post, the author transitions from discussing model files to working with actual numbers in PyTorch on the GPU.
A tensor is defined as a typed, multi-dimensional array that exists on a specific device, whether it be the CPU or the GPU. The author demonstrates how to create tensors, examine their properties such as shape and byte size, transfer them to the GPU, and assess the impact of precision on speed and memory usage. By the end of the post, the author aims to achieve proficiency in using CUDA, which involves selecting a device, transferring data to it, and accurately timing GPU work.
Everything is executed on the DGX Spark platform via SSH, with the author reusing the PyTorch environment from Week 1, located at "~/venvs/w1". This environment already includes a CUDA build of PyTorch.
The author introduces several key terms for different array dimensions: a scalar (a single number, Rank 0), a vector (a list of numbers, Rank 1), a matrix (a grid of numbers, Rank 2), and a tensor (the general term for any of these, including 3-D and higher dimensions). The rank of a tensor refers to the number of dimensions, while the shape denotes the size along each dimension, and dtype signifies the number format, such as FP32, FP16, or BF16. The device indicates where the tensor is located – either the CPU or the GPU.
To illustrate these concepts, the author creates a scalar, vector, matrix, and 3-D tensor, displaying each one's rank, shape, dtype, byte size, and device. The script also showcases three common operations: an element-wise addition (position by position), a broadcast (a smaller tensor applied across a larger one), and a host-to-device transfer (moving a tensor to the GPU).
All measurements of size and byte count in this script are derived from PyTorch, ensuring accuracy rather than relying on hand estimates.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.