The Einstein test: what happens when AI tries to rediscover relativity?

In 1915, Albert Einstein presented his general theory of relativity, forever altering our understanding of space-time and gravity. This groundbreaking theory, which explains gravitation as a deformation of space-time by mass, is a cornerstone of modern physics, influencing fields ranging from black hole research to gravitational wave measurements and guiding space missions and GPS satellites.

Consequently, general relativity has also become a benchmark for AI researchers, who are questioning whether their algorithms can replicate such a monumental discovery.

At the India AI Summit in New Delhi, Demis Hassabis, co-founder of Google DeepMind, suggested training a large language model (LLM) on all known scientific knowledge up to a specific date - 1911, the year before Einstein's theory was published - in an effort to gauge whether AI could reproduce general relativity. Hassabis argued that such a test would serve as a valuable gauge for evaluating artificial general intelligence (AGI), a long-sought goal in the AI industry.

While the idea of an "Einstein test" for AI may seem intriguing, early attempts to replicate this experiment have thus far yielded limited success. In December 2024, Owain Evans, a researcher at the non-profit Truthful AI, discussed "vintage" or "historical" LLMs - systems trained only on historical data up to a particular date.

These models were tasked with rediscovering scientific principles from history. However, their results thus far demonstrate more limitations than strengths, indicating that current AI is not well-equipped to make groundbreaking leaps in scientific understanding.

A preprint titled 'Can AI follow in Einstein's footsteps?' by Ido Kaminer, a researcher in quantum optics at Technion — Israel Institute of Technology, suggests that while AI might have the potential to rediscover relativity, it requires a fundamental shift in the way these models are built. Kaminer and his colleagues argue that Einstein's breakthrough wasn't a result of inductive reasoning, which derives general rules from accumulated data, but rather abductive reasoning: a creative leap that invents a cause for a singular phenomenon.

Current AI models, however, are primarily designed to perform statistical correlations and provide statistically likely answers based on known examples.

Moreover, researchers such as Sendhil Mullainathan, a computer scientist at MIT, have found that LLMs excel at predicting patterns in vast data sets but struggle to make the kind of abductive leaps necessary for paradigm shifts in understanding. For instance, when given synthetic data on various planetary systems obeying Newtonian mechanics, an orbital mechanics foundation model designed by Mullainathan and his team failed to infer the true law of gravitation but instead inferred a different law for each planetary system, each unique and incorrect.

Despite these challenges, recent advancements in AI have shown that these models can sometimes intuit logical structures underlying their training data. For example, an AI chatbot developed by OpenAI recently disproved an 80-year-old conjecture by Hungarian mathematician Paul Erdős, a phenomenon that can be considered a "genuine conceptual advance."

However, this achievement was not a result of brute force computation but rather the AI's ability to combine existing ideas in new ways, rather than an Einstein-like flash of discovery from seemingly nowhere.