Fractal patterns reveal hidden limits to AI predictability

Researchers at the University of Sydney have unveiled a fundamental mathematical reason behind artificial intelligence systems' varying behaviors, even when trained under nearly identical conditions. This discovery, published in Nature Machine Intelligence, could enhance the reliability and trustworthiness of AI technologies. As AI becomes ubiquitous in healthcare, business, research, and daily life, ensuring consistent and predictable performance poses a significant challenge.

The study, led by Professor Pulin Gong and Ph.D. student Andrew Ly, suggests that some AI unpredictability arises not from technical flaws or hardware inconsistencies, but from the underlying mathematical structure of how neural networks learn.

Traditional explanations attribute AI behavior differences to external sources of randomness, such as the order of training data or minor numerical variations from hardware. However, Gong's team found that even after accounting for these factors, minute differences in a model's starting conditions can lead to entirely different outcomes.

Neural networks, the backbone of today's advanced AI, learn by tweaking millions or billions of internal parameters during training. The Sydney researchers investigated why networks sometimes arrive at divergent solutions despite identical data, training procedures, and deterministic computing environments.

To illustrate this phenomenon, the researchers used the analogy of a map. Each possible starting point for training forms a location on this map, colored according to its eventual outcome. These outcomes group together in regions known as "basins of attraction." The team discovered that these basins exhibit intricate fractal structures called "riddled basins." In such landscapes, every point lies arbitrarily close to points leading somewhere else, with no clear distinction between regions, no matter how much you zoom in.

Even in fully deterministic environments, where identical training runs should yield identical results, the researchers found evidence of this fractal structure across various network architectures, datasets, and training methods. Simulating approximately 500,000 neural networks, they observed that altering the starting settings by a single binary digit (bit) sent training to a different internal configuration in nearly half of the cases (about 48%).

Even increasing the precision of a network's starting conditions by a factor of 10 billion produced no measurable improvement in reproducibility.

Understanding these riddled basins could guide the design of more dependable AI systems and reduce the associated costs and uncertainties. The researchers argue that unpredictability may represent a fundamental feature of how many neural networks operate, rather than a problem that can be entirely engineered away. They introduce new tools for measuring this sensitivity and propose future approaches, such as redesigning network architectures to reduce the mathematical conditions that give rise to instability.

As AI continues to permeate society, this research offers a new way to comprehend the limits of predictability and reproducibility in deep learning.