{
  "id": 15936411,
  "title": "A hot-expert cache made a 181 GiB model faster on one 96 GiB GPU",
  "url": "https://urgent.news/2026/10/21/a-hot-expert-cache-made-a-181-gib-model-faster-on-one-96-gib-gpu",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-21T19:22:03.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/pctablet505/a-hot-expert-cache-made-a-181-gib-model-faster-on-one-96-gib-gpu-4b03"
  },
  "original_language": "en",
  "account": "GLM-5.3-Flash model achieves faster performance on a GPU with limited memory by optimizing the placement of \"hot experts\" – the most frequently used components of the model during inference. The key findings are:\n\n1. The model, weighing 181 GiB, was run on a GPU with 96 GiB of memory. Writing speed peaked at 16.5 tokens per second, despite offloading more experts to the GPU, which slowed performance.\n\n2. By keeping the hot experts on the GPU, the writing speed increased to 16.50 tokens per second, a more than 1.5x improvement over the original 14.99 tokens per second.\n\n3. Reading speed also improved to 1,771 tokens per second for fresh prompts, a 2.9x enhancement compared to the original setup.\n\n4. The primary bottleneck was the PCIe transfer of individual experts for each token, consuming nearly all the GPU's time. A gather kernel that reads wide data blocks significantly increased the transfer rate to 52.1 GB/s, or 92% of the GPU's memory link.\n\n5. Offloading more experts (125 GiB instead of 105) to the GPU, and caching the hottest 66 experts in host RAM, resulted in the fastest performance at 16.50 tokens per second.\n\n6. Prefetching prompts in groups of 64 experts greatly accelerated reading, increasing from 614 to 1,750 tokens per second for long prompts.\n\nHowever, several optimization attempts failed, such as:\n\n- Implementing a recency (LRU) cache for hot experts, which only achieved a 9% hit rate.\n- Running experts on the CPU, which slowed performance to 0.895x the GPU's speed.\n- Sampling traffic to profile expert usage, which flattened the skew but did not improve performance.\n\nThe bug where the model emitted the same token forever was attributed to silent errors in two kernels computing in 4-bit precision, which could be resolved by using the Marlin kernel that unpacks weights first.\n\nIn summary, by strategically placing the most frequently used components of the neural network on the GPU's memory, this approach achieved substantial speed improvements without compromising the model's output.",
  "summary": "The build of GLM-5.3-Flash I run is 181 GiB, and my GPU has 96 GiB. It still writes up to 16.5 tokens a second on one desktop. Offloading more on its own made it slower; keeping each layer's hot experts on the card made it faster. One of the eight bugs on the way gave no error at all. The model just emitted the same token, forever. I will come back to that one. Each bar is its own measured setup.…",
  "key_points": [
    "GLM-5.3-Flash model optimized by placing hot experts on 96 GiB GPU",
    "Writing speed increased to 16.50 tokens per second, a 1.5x improvement",
    "Reading speed enhanced to 1,771 tokens per second, a 2.9x enhancement"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}