You could have invented transformers from memory

Large language models today can recall obscure facts from any corner of the world. This raises the question of how these models store such vast amounts of information. In this post, the author explores the idea of memory stores through the lens of linear maps, multi-layer perceptrons (MLPs), and self-attention mechanisms.

A memory store is a function that holds M memories, each consisting of a key-value pair. To query the store, an input x is used to recall the corresponding value. The ideal memory store should be able to recall the stored value with minimal interference from other memories. The interference-to-signal ratio is used to measure this, with the capacity of the store defined as the number of memories it can hold before interference equals the signal.

Linear layers are the simplest form of memory store, where memories are accumulated via outer products. These weights are updated using gradients during training, effectively serving as attention over all training inputs. However, the capacity of a linear store relies on its key dimension, which is limited by hardware constraints.

To improve capacity without increasing the key dimension, algorithmic enhancements can be made. By carefully selecting weights to reduce interference during recall, the performance can be enhanced. This leads to the development of MLPs and self-attention mechanisms, which both offer better recall and increased capacity. With 64-dimensional keys, the jump in capacity is from roughly 64 memories to about 2,700, and eventually surpasses one million memories.