Was TinyStories the Domain or the Vocabulary?
I have been running a long series of experiments on whether small models can be made to behave usefully, and one obvious thing to borrow is whatever TinyStories did. That paper trained very small transformers to write coherent children's stories, and the detail everyone remembers is the vocabulary: roughly 1,500 words. An earlier experiment of mine had already varied the domain at a fixed…
A series of experiments was conducted to determine whether small transformer models could be made to perform useful tasks. TinyStories, a prior study, trained small transformers to write coherent children's stories using a vocabulary of around 1,500 words, with improvements in perplexity and reduced repetition.
The natural follow-up question was to hold the domain constant and vary the vocabulary. It was hypothesized that limiting the vocabulary would improve performance. However, this approach had the opposite effect, leading to a 4-point decrease in scores and reduced quality of generated text. The research code is not publicly available, but the results were obtained directly from the experiment's harness and recorded data.
A smaller vocabulary made the next-token prediction easier, as everything outside the vocabulary was replaced with the "unk" placeholder. Consequently, any model with a smaller vocabulary would perform better on a naive score. To ensure fairness, the models were scored only on positions whose targets were within the smallest vocabulary, which contained 1,498 types. Identical positions and targets were used across all three models, resulting in 3,000 scored positions per model.
The experiment used the same corpus, model architecture, and budget per model. The vocabulary caps were set at 1,500, 4,000, and 8,000 types. The results showed that the vocabulary cap at 1,500 resulted in a 4-point decrease in scores and led to a model that generated less novel content, with more repetition and the use of the "unk" placeholder. At higher vocabulary caps (4,000 and 8,000), the generated text was more coherent and contained a greater number of novel 4-grams.
The key takeaway from this experiment is that restricting the vocabulary does not simplify the generative process; instead, it damages the representation of the content words, making the model generate less meaningful text. The benefit observed in TinyStories was due to the simplicity of the generated language, not the small vocabulary. The experiment also highlights the importance of scoring all positions, not just those within the limited vocabulary, to obtain accurate results.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
