Predicting the Speed of a 276B Model Streamed From an SSD
A mixture-of-experts model only activates a few of its experts per token, which means (unlike a dense model, where every forward pass touches every weight) you can leave the weights on disk and read in just the ones each token routes to. That turns "does this model fit in RAM" into "how fast is your SSD," and it puts models far larger than your machine nominally supports within reach, at some…
The mixture-of-experts (MoE) model's weight activation is limited to a few experts per token, unlike a dense model where every forward pass touches every weight. This enables the model to be larger than the machine's RAM, with the speed depending on the SSD's performance. To determine how fast a 276B Inkling-Small model runs, the author wrote a cost model to predict the on-disk size based on the model's configuration.
The model's prediction was validated by comparing it to a different model's published size, achieving a 1.1% accuracy. The total parameter count is nearly irrelevant in determining the per-token cost, which is instead set by the number of experts per token, the number of layers, and the size of each expert. A key finding was a cliff effect, where the fraction of experts fitting in 24 GB of RAM drops significantly beyond a certain model size.
To obtain a GB/s number for calculating tokens per second, the author benchmarked the SSD and corrected a previous misreading of a macOS API that caused an overestimation of the read bandwidth. The final predicted token rate for the Inkling-Small model running on a 24 GB Mac mini was 23 times faster than initially expected.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
