{
  "id": 4207332,
  "title": "Predicting the Speed of a 276B Model Streamed From an SSD",
  "url": "https://urgent.news/2026/08/29/predicting-the-speed-of-a-276b-model-streamed-from-an-ssd",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-29T14:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/megapixel99/predicting-the-speed-of-a-276b-model-streamed-from-an-ssd-50f8"
  },
  "original_language": "en",
  "account": "The mixture-of-experts (MoE) model's weight activation is limited to a few experts per token, unlike a dense model where every forward pass touches every weight. This enables the model to be larger than the machine's RAM, with the speed depending on the SSD's performance. To determine how fast a 276B Inkling-Small model runs, the author wrote a cost model to predict the on-disk size based on the model's configuration. The model's prediction was validated by comparing it to a different model's published size, achieving a 1.1% accuracy. The total parameter count is nearly irrelevant in determining the per-token cost, which is instead set by the number of experts per token, the number of layers, and the size of each expert. A key finding was a cliff effect, where the fraction of experts fitting in 24 GB of RAM drops significantly beyond a certain model size. To obtain a GB/s number for calculating tokens per second, the author benchmarked the SSD and corrected a previous misreading of a macOS API that caused an overestimation of the read bandwidth. The final predicted token rate for the Inkling-Small model running on a 24 GB Mac mini was 23 times faster than initially expected.",
  "summary": "A mixture-of-experts model only activates a few of its experts per token, which means (unlike a dense model, where every forward pass touches every weight) you can leave the weights on disk and read in just the ones each token routes to. That turns \"does this model fit in RAM\" into \"how fast is your SSD,\" and it puts models far larger than your machine nominally supports within reach, at some…",
  "key_points": [
    "MoE model's weight activation limited to few experts per token",
    "276B Inkling-Small model's speed depends on SSD performance",
    "Predicted token rate 23 times faster than initially expected"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}