“We were looking at it wrong”: The explosive claim that’s upended the AI world
“I think I have good news about the nature of neural networks.” With those simple words of optimism, Nick Cammarata, a researcher at the AI startup Goodfire, kicked off a 26,000-word blog post that has knocked the AI world on its head, sparking wild speculation as well as cautious (and not-so-cautious) celebration. Cammarata’s claim is…
Nick Cammarata, a researcher at AI startup Goodfire, declared in a lengthy blog post that artificial neural networks may be more comprehensible than previously believed, causing a significant stir in the AI community. The post, which has garnered attention from researchers worldwide, has challenged the notion that large language models (LLMs) are fundamentally opaque and uncontrollable.
If true, this could be a pivotal moment in AI research, shedding light on the inner workings of these complex systems and potentially enabling better monitoring and control.
Cammarata's claim hinges on the idea that, despite the apparent entanglement of neurons within neural networks, there may be underlying patterns and logic that can be discerned with more diligent analysis. His perspective is rooted in the field of mechanistic interpretability, which aims to understand how neural networks function by "picking apart" their components, much like neuroscientists study brain activity using MRI scans.
Cammarata had previously worked at OpenAI, contributing to the development of tools that allowed researchers to probe the inner workings of AI models. In 2020, while examining the Inception v1 model, he observed instances of superposition, where a single neuron seemed to represent multiple concepts simultaneously - for example, being able to recognize both cars and cats. This phenomenon suggested that neural networks might be utilizing neurons to multitask, packing diverse information into a single unit.
Initially, Cammarata's findings led him to believe that neural networks were inherently chaotic and intractable, a view that fostered a confirmation bias in his approach to studying these models. However, as he delved deeper into the neural network structures, he gradually discovered that the initial pessimism was misplaced. By reexamining a seemingly nonsensical neuron responsible for recognizing cars and cats, Cammarata identified a common pattern—a V-shaped relationship between the two concepts.
This revelation marked a turning point in his understanding, leading him to reconsider the fundamental assumptions about neural networks.
Written by urgent.news from MIT Technology Review's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
