{
  "id": 6693981,
  "title": "The Embedding Table Was 72% of the Model",
  "url": "https://urgent.news/2026/09/11/the-embedding-table-was-72-of-the-model",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-11T03:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/megapixel99/the-embedding-table-was-72-of-the-model-2ghm"
  },
  "original_language": "en",
  "account": "The study found that 72% of a transformer model's size is taken up by an embedding table, which is essentially a lookup table. This led to the question of whether quantising the embedding table independently of the transformer body could save memory. The research showed that int4 quantisation per-token reduced the model's size by 6.2x while maintaining the same performance as the original fp32 model. This was due to the fact that the embedding table had large variations in norms between frequent and rare words. The research also found that per-row scales were more beneficial than per-tensor scales for the embedding table.",
  "summary": "An earlier experiment in this series had established something slightly deflating about a small transformer: quantising the whole network to int8 is free, and the bytes you save are better spent on count tables than on network precision. That is a useful result and it invites a sharper question, which is where the bytes actually were. tok.weight (embedding) 1,024,000 params 71.7% everything else…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}