A model doesn't read text: what a tokenizer decides for you
Originally published at cchinchilla.dev . Part 1 of From code to weights, a 12-part series on ML fundamentals for engineers. A language model has never seen a letter. It sees integers. Somewhere between your string and the first matrix multiply there's a function that turns text into a list of ids, and that function was frozen the day the model was trained. It isn't preprocessing. It's part of…
A language model does not perceive individual characters; instead, it interprets integers. Between the string input and the initial matrix multiplication, there exists a function that transforms text into a list of ids. This function is set during the model's training phase and remains unchanged thereafter. This process is not preprocessing, but rather an integral part of the model's structure.
The tokenizer, which performs this function, was determined when the model was initially trained and cannot be altered afterward. This is the primary focus of this article, the first in a series aimed at engineers transitioning into ML engineering.
In the realm of compilers, the parser is a component that can be modified and adjusted over time. However, in the context of a language model, the tokenizer is a fixed element that was initially chosen before the model's training commenced. The tokenizer, similar to a lexer in compilers, takes characters and outputs a sequence of symbols, such as identifiers, keywords, and literals.
The primary objective of this tokenizer is to reduce the string into the minimum number of ids, using a predetermined vocabulary, which will then be used by the model. There are various strategies to achieve this, each with its own advantages and disadvantages.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.