How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Specializing the vocabulary of small models can reduce the cost of computation by up to 77% in real-world applications, according to preliminary results from the VocabX paper. The issue arises because the tokenizer used in general-purpose models, such as Qwen and Gemma, is trained on a wide variety of texts, often splitting domain-specific language into multiple tokens.

This results in fragmentation, where common domain phrases like JSON envelopes or boilerplate clauses are broken down into several tokens, and excess, where the model must compute a score for irrelevant language entries it will never use. The inference process for these models involves two phases: prefill, where the model processes input in a single pass, and decode, where it generates the response one token at a time.

The latter phase is particularly memory-bound, and every extra token costs more. In the case of an automated Portuguese essay grader built on the Qwen3-4B model, the original tokenizer had 151,669 entries, many of which were irrelevant to the task. By using a new method called VocabX to specialize the vocabulary, the number of entries was reduced from 152,000 to 65,000, resulting in a 77% cost reduction without altering the base model.