A researcher says modern neural networks are far more interpretable than believed, with most neurons understandable in models up to trillions of parameters (Nick Cammarata/Goodfire)
The cause of polysemanticity in early model organisms wasn't superposition, but several ordinary, mostly fixable issues.
Neural networks may be more interpretable than previously thought, according to research by a researcher named Nick Cammarata. After months of studying the inner workings of image and language models, Cammarata believes that most neurons in networks with up to trillions of parameters can now be understood. This challenges the notion that neural networks are inherently tangled and difficult to comprehend.
The research suggests that the initial belief in superposition, where neural networks group unrelated concepts together, was misguided. Instead, Cammarata proposes that the cause of polysemanticity, or neurons with multiple unrelated jobs, is likely due to architectural problems. These issues cause later layers to overflow and become less interpretable. However, recent improvements in network architecture have led to a significant increase in the interpretability of even the final layers.
Cammarata outlines two approaches to mechanistic interpretability: rapid normal science, which involves directly studying the network's neurons and circuits, and the second approach, which starts with a list of hypothesized features and attempts to extract them all. The field initially focused on the first approach, but it later shifted to the second due to the perceived complexity of neural networks.
The research argues that the canonical examples of polysemanticity and superposition were misread and that the outlook for mechanistic interpretability is far better than previously thought.
The study applied this analysis to several models, including InceptionV1 and modern language models. In the case of InceptionV1, 99-100% of neurons in the final layer became easily interpretable after implementing ordinary improvements. Similarly, when studying Anthropic's language models, the median neuron received a score of 0 out of 14, indicating an inability to form a hypothesis.
However, with better tools and modern models, the situation appears more promising, with a significant percentage of neurons showing a primary job or a combination of related jobs.
Written by urgent.news from Techmeme's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
