{
  "id": 15771618,
  "title": "Tokenization is why your agent bill looks weird",
  "url": "https://urgent.news/2026/10/20/tokenization-is-why-your-agent-bill-looks-weird",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T17:15:47.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/dev_jhawar/tokenization-is-why-your-agent-bill-looks-weird-4n47"
  },
  "original_language": "en",
  "account": "Tokenization is the reason your agent bill might appear strange. While models read integers, there's a crucial intermediary step: a tokenizer. This component takes your string input and divides it into pieces from a predefined vocabulary, typically ranging from 30,000 to 200,000 entries. Each piece is then assigned an integer ID. Common words become one token, while rarer words are broken down into smaller tokens.\n\nUsing the encapsulation library tiktoken, you can observe this process firsthand. For example, the word \"tokenization\" is divided into two tokens, whereas a rare word like \"Pneumonoultramicroscopicsilicovolcanoconiosis\" is split into seventeen tokens. Even a leading space can alter the tokenization, affecting how prompts behave differently based on whitespace.\n\nTokenizers are built using various methods, most notably Byte Pair Encoding (BPE). BPE starts with individual characters, identifies the most frequent adjacent pair, merges them into a new symbol, and repeats the process. By iterating over a vast corpus of text—merging tens of thousands of pairs—you end up with a concise vocabulary. The more common words in the training data are assigned shorter tokens, while less frequent words receive longer tokens.\n\nDifferent tokenizers employ distinct strategies, such as GPT-2 and OpenAI's tiktoken, which use BPE; Byte-level BPE, which handles any character without limitations; WordPiece, which selects merges based on likelihood; and SentencePiece, which processes raw text including spaces, making it suitable for languages without clear word boundaries like Hindi.\n\nWhen comparing token counts across four major tokenizers—GPT-2, cl100k_base (representative of GPT-4 era), o200k_base (representative of GPT-4o era), and T5's SentencePiece—several patterns emerge. English texts generally perform consistently across all tokenizers due to the extensive training data in English. In contrast, code and JSON significantly reduce in token count, likely because newer vocabularies have been trained on more technical content, merging whitespace and common JSON punctuation into fewer tokens.\n\nHindi presents a notable challenge. The same sentence, when tokenized with older encodings, uses 111 tokens, whereas newer encodings like o200k reduce it to just 24 tokens. This 4x difference in cost and context usage can lead to poorer performance and increased computational requirements. Similarly, emojis exhibit inconsistent tokenization across tokenizers, complicating predictions for non-English users who may pay more in terms of both cost and context usage.\n\nThese discrepancies can significantly impact the cost and functionality of language models in tools, such as agent bills. For instance, a JSON schema of 610 characters translates to approximately 160 tokens, while tool schemas for agents can quickly accumulate, leading to substantial token overhead. Compact representations, like JSON with minimized whitespace, help mitigate these costs, but careful consideration of the tokenization process is crucial to avoid unexpected expenses and limitations.\n\nTo gain a deeper understanding, experiment with your own strings, languages, and tool schemas. Swapping in your own data and observing the tokenization results can reveal surprising insights, highlighting the importance of using the appropriate tokenizer for your specific model and text type.",
  "summary": "Models don't read text. They read integers. Between your string and the model there's a tokenizer, and it quietly decides how much you pay, how much fits in the context window, and why your model is weirdly bad at counting letters. I always treated it as a preprocessing detail. Then i started counting. What a tokenizer actually does It chops text into pieces from a fixed vocabulary, usually 30k…",
  "key_points": [
    "Tokenization divides strings into vocabulary pieces from 30k-200k entries",
    "Byte Pair Encoding (BPE) merges frequent character pairs into new symbols",
    "Hindi tokenization differs significantly from other languages, using 24 tokens vs 111"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}