Transformers Explained Visually

Transformers represent a revolutionary neural network architecture that has transformed the field of Artificial Intelligence. Introduced in 2017's paper "Attention is All You Need," Transformers have since become the foundation for deep learning models powering text-generative AI systems such as OpenAI's GPT, Meta's Llama, and Google's Gemini. Their utility extends well beyond text, including audio generation, image recognition, protein structure prediction, and game playing, showcasing their broad applicability.

At the heart of text-generative Transformer models lies the concept of next-token prediction: given a user's input text, the model predicts the most probable next token (a word or subword). The crux of Transformers' power is their self-attention mechanism, enabling them to handle entire sequences and capture long-range dependencies more efficiently than earlier designs.

A prominent example of a text-generative Transformer is the GPT-2 family of models, which utilize the GPT-2 (small) model with 124 million parameters. While not the latest or most powerful model, it retains many essential architectural components, making it an excellent entry point for grasping foundational concepts.

The process of generating text with a Transformer begins with a user's prompt, such as "Data visualization empowers users to." This input must be converted into a numerical format compatible with the model. This transformation occurs through a process called embedding, which involves tokenizing the input, obtaining token embeddings, adding positional information, and combining token and position encodings to create the final embedding.

Tokenization breaks down the input text into smaller units called tokens, which can be individual words or subwords. For instance, "Data" and "visualization" are single tokens, while "empowers" is split into two tokens. A predefined vocabulary, in the case of GPT-2 (small), consists of 50,257 unique tokens. These tokens are then represented as vector representations in a matrix with dimensions based on the model (e.g., GPT-2 (small) uses 768-dimensional vectors).

This extensive matrix allows the model to convey semantic meaning, as tokens with similar usage or meaning are placed closer together in the high-dimensional space.

Positional encoding, another crucial aspect of Transformers, encodes the position of each token in the input prompt. GPT-2 (small) independently trains its positional encoding matrix, which is integrated during training. This information helps the model understand the sequence of tokens, a critical aspect of processing language.

The Transformer's core processing unit is the Transformer block, composed of multi-head self-attention and a Multi-Layer Perceptron (MLP) layer. The block is stacked sequentially to build up intricate representations of the input. In the GPT-2 (small) model, this block structure is repeated twelve times.

Self-attention, the key mechanism within the Transformer block, allows the model to capture relationships among tokens in a sequence. This capability ensures that each token's representation is influenced by the others, leading to higher-order representations of the input. Multiple attention heads within the self-attention mechanism enable the model to consider these relationships from various perspectives. For example, one head may focus on short-range syntactic links, while another captures broader semantic context.

To illustrate these concepts, consider the QKV vectors derived from input embeddings. These vectors, derived by multiplying the input embedding matrix with learned weight matrices for Q (query), K (key), and V (value), facilitate attention score calculations. These scores determine how much focus each token should receive when generating predictions.

In GPT-2 (small), these vectors are split into 12 attention heads, each processing a segment of the embeddings independently and capturing different syntactic and semantic relationships. This design allows for parallel learning of diverse linguistic features, enhancing the model's representational power.