Is ‘Attention’ the Essence of Intelligence?

Everything that has swept the world since the release of ChatGPT in 2022—ranging from a burgeoning AI economy to data centers, chatbots serving as friends, doctors, and mathematicians, and discussions of existential risk—originates, to some extent, with a young man named Dzmitry Bahdanau in a University of Montreal lab. In 2014, the 23-year-old Belarusian, who had been coding competitively since childhood but had no prior academic research experience, was tasked with helping a computer translate English sentences to French.

At the time, artificial neural networks were already demonstrating promise in pattern recognition tasks like handwriting recognition and object identification in photos. These networks learn by strengthening or weakening connections between nodes based on training data. Although Bahdanau harbored doubts about their potential to achieve human-level intelligence, his adviser, computer scientist Yoshua Bengio, and a postdoctoral researcher, Kyunghyun Cho, were convinced.

Translation posed a significant challenge due to the complex grammatical structures and idioms present in natural languages. Unlike simple word-for-word swapping, translation required capturing the nuances and subtleties of language. Researchers had struggled to achieve satisfactory results, as the performance of neural networks declined with longer sentences due to memory limitations.

The solution lay in a different network architecture. Bahdanau introduced a mathematical operation called "attention," allowing the network to handle long sentences as effectively as short ones. This approach proved scalable, and other AI researchers quickly incorporated attention into their neural network designs. In 2017, a team at Google introduced a transformative architecture called transformers, which enhanced the attention mechanism and escalated performance while consuming less computational power.

The groundbreaking paper, "Attention Is All You Need," reported that transformers achieved state-of-the-art translation capabilities and has since been cited over 250,000 times, marking a watershed moment in AI. This attention mechanism is at the heart of understanding how a single AI model can engage in conversation, diagnose diseases, generate realistic videos, solve complex mathematical problems, and more. Essentially, is attention the core of intelligence—both artificial and human?