Is AI reasoning right for the wrong reasons?

AI reasoning has been a hot topic in the past few years, especially with the emergence of large reasoning models (LRMs). These models are trained to produce chains of thought, a method where the AI generates intermediate steps before arriving at an answer to complex queries. However, the idea of AI reasoning has been met with skepticism and criticism, with some researchers claiming it is an "illusion of thinking" and subject to "complete accuracy collapse" under certain conditions.

Despite this initial skepticism, LRMs have been making significant advancements in recent months. In May 2026, an OpenAI model reportedly solved a famous open mathematical research problem in one shot. Additionally, Google DeepMind and mathematician Terence Tao used AI to rediscover or improve solutions to 67 problems spanning various fields of mathematics. These achievements have raised questions about the true nature of AI reasoning and whether it is as reliable and generalizable as claimed.

However, recent research has cast doubt on the authenticity of AI reasoning. A study by Santa Fe Institute found that LRMs can perform well on reasoning benchmarks by using "surface-level 'shortcuts'." This suggests that the AI systems may not be demonstrating real reasoning but instead exploiting specific patterns or "shortcuts" to achieve the desired results.

Furthermore, research has shown that the chains of thought generated by LRMs may not be a faithful representation of the AI's inner workings. These "intermediate tokens" may appear to be meaningful intermediate steps, but they could be incidental to the actual reasoning process. In some cases, replacing these tokens with incorrect or irrelevant information did not degrade the model's performance on reasoning tasks.

Additionally, studies have demonstrated that even meaningless filler tokens could function effectively in place of a human-readable chain of thought.

So, can AI reasoning be both BS (meaningless or unreliable) and not at the same time? The answer appears to be yes. The existence of these "surface-level shortcuts" and the questionable reliability of the AI's "thinking process" suggests that AI reasoning may not always be dependable. However, the fact that LRMs can achieve remarkable feats in certain domains indicates that there is some merit to the idea of AI reasoning, even if it may not be as robust or generalizable as initially thought.

In conclusion, the state of AI reasoning remains a complex and puzzling issue. While LRMs have demonstrated impressive abilities, the underlying mechanisms and reliability of their reasoning processes are still subjects of debate and research. As AI continues to advance, it is crucial to maintain a critical perspective on the claims surrounding AI reasoning and to remain vigilant about the potential limitations and pitfalls.