Fully Interpretable Minimal Transformers: From Geometry to Algorithm
We present a framework for building and interpreting minimal transformer models. By constraining a transformer's embedding dimension and head size to 2, we enable full two-dimensional visualization of its internal representations. Embeddings, query/key/value transforms, attention outputs, residual streams, and decision boundaries can all be seen directly. Our central claim is that the learned…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.