{
  "id": 19108747,
  "title": "Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide",
  "url": "https://urgent.news/2026/11/09/tokenization-bpe-from-scratch-vocab-size-and-what-tokenizers-hide",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-09T07:01:46.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ali_82f3409ed625e8ba8515b/tokenization-bpe-from-scratch-vocab-size-and-what-tokenizers-hide-23mn"
  },
  "original_language": "en",
  "account": "Tokenization is the process of breaking down text into smaller units called tokens before a language model processes them. These tokens are usually integers that represent words or parts of words. The tokenizer is what the model interacts with, not the original letters or words. To construct a tokenizer, we can use a method called byte pair encoding (BPE).\n\nBPE begins with individual characters and gradually merges the most frequently occurring adjacent character pairs into new tokens. This process continues until the vocabulary reaches a predetermined size. Larger vocabularies can represent text more efficiently, saving both tokens and contextual information, but they also require more parameters in the embedding matrix, which incurs additional computational costs.\n\nThe vocabulary, a list of tokens, is learned from data, which means it reflects the biases present in the data it was trained on. For example, English tends to have a smaller vocabulary, while languages like Urdu can require larger vocabularies. Furthermore, not all tokens are created equal, as some are trained more extensively than others.\n\nPhilip Gage first described BPE as a data compression technique in 1994. It wasn't until 2016 that Rico Sennrich, Barry Haddow, and Alexandra Birch adapted it for neural machine translation in their paper, Neural Machine Translation of Rare Words with Subword Units. This adaptation allowed any word to be represented as a sequence of frequent pieces, greatly improving the model's ability to handle rare words.\n\nTo better understand BPE, we implemented the algorithm in Python and applied it to a small corpus of words from our own published articles. After ten merges, we saw that the most frequent words became whole tokens, while rare words were broken into smaller pieces. For instance, \"low\" remained as one token, while \"newest\" became \"low + e + s + t\" and \"wider\" became \"wid + er\".\n\nWe then trained our BPE tokenizer on 83,159 real words from a larger corpus and measured its performance. The results showed that the first 100 merges cut the token count in half, and by the 2,000th merge, we achieved a token representation of 4.60 characters per token. However, beyond 5,000 merges, adding more tokens didn't provide any additional benefits for the held-out text, as the extra merges mostly memorized the quirks of our training corpus rather than learning general patterns.\n\nIn practical applications, BPE is a crucial component of many language models, including GPT-2 and Llama 3. It efficiently turns text into tokens, allowing the model to focus on the content rather than the individual letters or words.",
  "summary": "Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide Before a language model reads a single word, a tokenizer has already decided how that word gets cut up. The model never sees letters. It never sees words either. It sees token IDs, numbers drawn from a fixed list, and everything the model can and cannot do starts with that list. We built byte pair encoding from scratch, trained…",
  "key_points": [
    "Tokenization breaks text into smaller units called tokens.",
    "BPE merges frequent character pairs into new tokens until reaching a predetermined vocabulary size.",
    "BPE is used in language models like GPT-2 and Llama 3 to efficiently represent text as tokens."
  ],
  "editors_take": "Balancing vocabulary size against computational costs forces developers to navigate a trade-off between efficient text representation and the risk of memorizing training data quirks instead of general linguistic patterns.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}