Don't be fooled–LLMs don't reason

In March 2016, the author observed a Go match between AlphaGo and Lee Sedol, one of the world's best Go players. Despite appearing absurd to observers, AlphaGo's move two by move 37 ultimately won the game. Lee Sedol remarked that he initially believed AlphaGo relied solely on probability calculations, but the creative move changed his mind, suggesting that AlphaGo was indeed creative.

While Deep Blue, the chess-playing supercomputer, achieved victory by calculating ahead six to eight moves per player and evaluating 200 million chess positions per second, Go's complexity posed an even greater challenge. AlphaGo's remarkable intuition was not a result of pure probability; rather, it combined intuitive leaps with deliberate search, examining thousands of possible game branches to weigh the long-term consequences.

This distinction between intuition and deliberation mirrors the human cognitive model of System 1 and System 2 thinking. Unlike contemporary AI models like large language models, which generate text through a system 1 process involving rapid pattern completion, AlphaGo possessed a separate reasoning mechanism. It maintained an epistemic state—a record of what it knew, doubted, and ruled out throughout the reasoning process.

This comprehensive representation of its knowledge allowed AlphaGo to synthesize information and make informed decisions. Unfortunately, today's AI models lack this explicit reasoning infrastructure. They generate answers by iterating the next-token prediction process, often without maintaining a coherent, verifiable epistemic state.

This limitation poses significant challenges in critical applications where understanding the reasoning process is crucial, such as in medicine, engineering, and scientific research. The author, reflecting on these insights, decided to leave Google DeepMind and advocate for a new approach to machine reasoning, drawing inspiration from AlphaGo's architecture.