We tested "tokenize before you compress" against 452 configurations, and it mostly held up
A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text by tokenizing it into subwords instead of raw bytes, why don't we tokenize text before handing it to a byte-level compressor like LZMA or zstd? It felt like the kind of idea someone must have already tried and quietly dropped. So instead of writing a…
Researchers @u84u and @ronak-create conducted an experiment to test if tokenizing text before compressing it with byte-level compressors like LZMA or zstd would result in smaller compressed output. They built a harness called parmar to test various configurations on a large corpus. The hypothesis was that tokenizing text with BPE would make the token stream roughly 45% smaller than the original text, allowing compressors to cover more source text with their limited dictionary windows.
However, the benefits of pre-tokenization only became apparent once the corpus was larger than the compressor's window. The study found that pre-tokenization only provided a size advantage for compressors with larger windows (like LZMA and zstd), and the advantage plateaued after reaching a certain corpus size. Bzip2 was not a suitable backend for this approach, as tokenization destroyed the byte-level structures the algorithm exploited.
Multithreading interactions also impacted the results, with xz benefiting from threading more than zstd. Overall, the experiment showed that pre-tokenization can be a useful technique for compressors with larger windows, but its benefits are limited by the corpus size and the specific compressor used.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
