The physics of novel computation: beyond bits and chips to spikes and spins

Modern computation has become an indispensable tool in nearly every aspect of human life. Digital computers rely on billions of transistors etched onto silicon chips, functioning as tiny switches that represent binary values of 0 or 1. These transistors enable computers to perform complex tasks, from powering personal devices to running powerful supercomputers that can simulate intricate scientific phenomena.

However, the limits of modern computation are beginning to show, as the pursuit of faster and more efficient computing methods becomes increasingly challenging.

As transistors became smaller and more densely packed, the concept of "Moore's Law" – which predicted a doubling of transistors on a chip roughly every two years – held true for many years. This exponential increase in transistors led to exponential gains in computing power and speed. However, as transistors continued to shrink, they began to consume more power and generate more heat, causing clock speeds to plateau at around 3-5 GHz.

To overcome this limitation, researchers have turned to parallel processing techniques, such as those employed in graphics processing units (GPUs). GPUs feature hundreds to thousands of simpler cores, designed to handle multiple data elements simultaneously, making them ideal for tasks like image rendering and matrix multiplication.

The rise of artificial intelligence, particularly large language models (LLMs) like ChatGPT, has highlighted the need for further advancements in computation. LLMs are trained on vast amounts of data, such as trillions of tokens from various sources, to learn statistical patterns and predict the next token in a sequence. Training these models requires massive computational power, often employing thousands of GPUs and consuming significant electrical energy.

Additionally, inference processing, where a trained model generates responses to user prompts, is also power-intensive but operates continuously to serve users across the globe.

The environmental impact of this increased computational demand cannot be ignored. By 2030, global data centers, primarily driven by AI applications, are projected to consume around 1000 terawatt-hours (TWh) of electricity annually, equating to roughly 100 gigawatts of power. This surge in energy consumption necessitates extensive cooling systems, leading to substantial water usage and potential strain on local power grids.

Consequently, there is a growing need to rethink the very foundations of computation, seeking alternative methods to overcome the physical constraints that currently limit the advancement of modern computing.