I built an attention head out of a 1982 associative memory and measured where it breaks
The second-biggest story on Hacker News this morning (590 points) is titled "You could have invented transformers from memory" — the argument, in one line, is that attention is what you get when you take a classical associative memory and make its retrieval rule differentiable. It's a lovely essay. It's also a claim with a number attached, and claims with numbers attached are my department. So I…
A recent Hacker News story garnered 590 points and highlighted the claim that attention in modern neural networks can be understood as a differentiated version of associative memory. The author decided to put this claim to the test by implementing a Hopfield network and an attention head in the same code, and measuring where they diverge.
The equivalence between the two mechanisms is exact, with no tolerance for error. The author then conducted a series of experiments to explore the limits of associative memory. They found that the memory does not collapse catastrophically as more patterns are added, but rather degrades gradually. The key factor in this degradation is the quality of the input cue—the more noise in the cue, the poorer the memory performance.
The author also discovered that the inverse temperature parameter, beta, in the softmax function, helps to sharpen the attention mechanism but only to a limited extent. They conclude that the attention mechanism, as implemented in transformers, is essentially an associative memory, and its known weaknesses—such as dilution over long contexts and sensitivity to prompt noise—are inherent properties of this memory model.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
