A Mathematical Framework for Transformer Circuits (2021)

Transformer language models are a rapidly emerging technology with wide-ranging real-world applications, such as GPT-3, LaMDA, Codex, Meena, and Gopher. However, as these models become more complex and powerful, they also become more prone to unexpected and potentially harmful behaviors. Despite years of training, creators and users frequently uncover capabilities within these models that they were unaware of beforehand.

One promising approach to tackling this issue is through mechanistic interpretability, a technique that attempts to reverse engineer the intricate computations performed by transformers, similar to how a programmer might reverse engineer complex code to understand its underlying logic. If successful, this method could provide a systematic way to explain current safety concerns, identify new issues, and potentially predict the safety challenges of future, more advanced models that have yet to be developed.

In a recent project, researchers attempted to reverse engineer vision models but have yet to achieve similar results for transformers or language models. In this paper, they take preliminary steps towards reverse engineering transformers by starting with the simplest possible models and gradually moving up from there. Their goal is to identify simple algorithmic patterns, motifs, or frameworks that can later be applied to larger and more complex models.

Specifically, they focus on transformers with two layers or less that contain only attention blocks, in contrast to larger, modern transformers like GPT-3, which have 96 layers and alternate attention blocks with MLP (multi-layer perceptron) blocks. By conceptualizing the operation of transformers in a mathematically equivalent but non-standard way, the researchers are able to gain significant understanding of how these small models function internally.

One notable finding is that specific attention heads, which they term "induction heads," can explain the in-context learning observed in these small models. These induction heads only emerge in models with at least two attention layers. The authors also provide examples demonstrating these induction heads in action on specific data.

While they do not apply their insights to larger models in this first paper, they plan to show that their mathematical framework and the concept of induction heads can still be relevant for much larger and more realistic models in a forthcoming paper. However, they acknowledge that they are still far from being able to fully reverse engineer such sophisticated models.

The study of transformers requires a new approach due to their complex architecture. Mechanistic interpretability involves breaking down models into human-interpretable components. To facilitate reasoning about the model, the researchers reframe transformers in unconventional ways. In modern deep learning, computational efficiency is a primary focus, and mathematical descriptions of models often reflect decisions made to optimize code efficiency.

However, when multiple representations of the same computation exist, the most human-interpretable representation may differ from the most computationally efficient one. The review of transformers also helps establish consistent terminology and introduce necessary notation, which is described in detail in an accompanying appendix.

To simplify the process and focus on the core ideas, the researchers analyze toy transformers with some simplifications, specifically attention-only transformers that lack MLP layers. This significant simplification allows for a more elegant treatment of the unique challenges posed by attention heads compared to the more complex Distill Circuits work.

Additionally, they have had less success in understanding MLP layers, so they dedicate more discussion to transformers with just attention blocks. Throughout the paper, the authors also make several minor changes for clarity and simplicity, including the omission of biases, which can be simulated by incorporating them into the weights.

Despite these simplifications, the researchers still discuss transformers with MLP layers in later sections.