Why Does a 125B AI Model Use Only 6B Parameters at a Time?

A new AI model has been released, boasting 125 billion parameters but only activating approximately 6 billion at any given moment. This raises the question: how can a model with such a large number of parameters only use a fraction of them? The answer lies in the architecture of modern AI models, specifically a concept known as Mixture of Experts (MoE).

In an MoE-based model, the massive parameter count is divided into specialized groups, called "experts." Each expert is a smaller neural network with its own set of parameters. The model also includes a "router" that decides which experts should process a particular input token. This router determines the optimal subset of experts to engage for a given token, allowing only a fraction of the total parameters to be active at any moment.

This approach enables the model to handle complex tasks without needing to activate all 125 billion parameters simultaneously. Instead, it activates only the relevant experts for each token, which could range from 6 billion to all 125 billion, depending on the specific task and the input. This selective activation allows for more efficient processing and resource utilization, making it possible for large-scale models to operate effectively.