Tokenization is why your agent bill looks weird

Tokenization is the reason your agent bill might appear strange. While models read integers, there's a crucial intermediary step: a tokenizer. This component takes your string input and divides it into pieces from a predefined vocabulary, typically ranging from 30,000 to 200,000 entries. Each piece is then assigned an integer ID. Common words become one token, while rarer words are broken down into smaller tokens.

Using the encapsulation library tiktoken, you can observe this process firsthand. For example, the word "tokenization" is divided into two tokens, whereas a rare word like "Pneumonoultramicroscopicsilicovolcanoconiosis" is split into seventeen tokens. Even a leading space can alter the tokenization, affecting how prompts behave differently based on whitespace.

Tokenizers are built using various methods, most notably Byte Pair Encoding (BPE). BPE starts with individual characters, identifies the most frequent adjacent pair, merges them into a new symbol, and repeats the process. By iterating over a vast corpus of text—merging tens of thousands of pairs—you end up with a concise vocabulary. The more common words in the training data are assigned shorter tokens, while less frequent words receive longer tokens.

Different tokenizers employ distinct strategies, such as GPT-2 and OpenAI's tiktoken, which use BPE; Byte-level BPE, which handles any character without limitations; WordPiece, which selects merges based on likelihood; and SentencePiece, which processes raw text including spaces, making it suitable for languages without clear word boundaries like Hindi.

When comparing token counts across four major tokenizers—GPT-2, cl100k_base (representative of GPT-4 era), o200k_base (representative of GPT-4o era), and T5's SentencePiece—several patterns emerge. English texts generally perform consistently across all tokenizers due to the extensive training data in English. In contrast, code and JSON significantly reduce in token count, likely because newer vocabularies have been trained on more technical content, merging whitespace and common JSON punctuation into fewer tokens.

Hindi presents a notable challenge. The same sentence, when tokenized with older encodings, uses 111 tokens, whereas newer encodings like o200k reduce it to just 24 tokens. This 4x difference in cost and context usage can lead to poorer performance and increased computational requirements. Similarly, emojis exhibit inconsistent tokenization across tokenizers, complicating predictions for non-English users who may pay more in terms of both cost and context usage.

These discrepancies can significantly impact the cost and functionality of language models in tools, such as agent bills. For instance, a JSON schema of 610 characters translates to approximately 160 tokens, while tool schemas for agents can quickly accumulate, leading to substantial token overhead. Compact representations, like JSON with minimized whitespace, help mitigate these costs, but careful consideration of the tokenization process is crucial to avoid unexpected expenses and limitations.

To gain a deeper understanding, experiment with your own strings, languages, and tool schemas. Swapping in your own data and observing the tokenization results can reveal surprising insights, highlighting the importance of using the appropriate tokenizer for your specific model and text type.