Quadrillion-parameter transformers and how to train them

Recent research explores training models with billions to trillions of parameters, highlighting a vast gap in sample efficiency compared to human brains. Extrapolating existing scaling laws to such massive models suggests needing quintillions of parameters on datasets of 30 trillion tokens. The study finds that as datasets stay fixed, the optimal parameter count continues growing with compute, and performance improves with longer training across multiple epochs.

The researchers develop a new scaling law incorporating ensemble methods, showing that increasing ensemble members yields further loss reduction without additional compute. By fitting a comprehensive model that accounts for factors like unique tokens, model size, epochs, and ensemble members, their approach achieves near-optimal performance.

This work suggests the need to train increasingly larger models over longer periods to achieve optimal efficiency in a data-constrained regime driven by growing compute resources.