{
  "id": 2272845,
  "title": "We tested \"tokenize before you compress\" against 452 configurations, and it mostly held up",
  "url": "https://urgent.news/2026/08/21/we-tested-tokenize-before-you-compress-against-452-configurations-and",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-21T01:37:16.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ronak_parmar_033c50d168b5/we-tested-tokenize-before-you-compress-against-452-configurations-and-it-mostly-held-up-4m6p"
  },
  "original_language": "en",
  "account": "Researchers @u84u and @ronak-create conducted an experiment to test if tokenizing text before compressing it with byte-level compressors like LZMA or zstd would result in smaller compressed output. They built a harness called parmar to test various configurations on a large corpus. The hypothesis was that tokenizing text with BPE would make the token stream roughly 45% smaller than the original text, allowing compressors to cover more source text with their limited dictionary windows. However, the benefits of pre-tokenization only became apparent once the corpus was larger than the compressor's window. The study found that pre-tokenization only provided a size advantage for compressors with larger windows (like LZMA and zstd), and the advantage plateaued after reaching a certain corpus size. Bzip2 was not a suitable backend for this approach, as tokenization destroyed the byte-level structures the algorithm exploited. Multithreading interactions also impacted the results, with xz benefiting from threading more than zstd. Overall, the experiment showed that pre-tokenization can be a useful technique for compressors with larger windows, but its benefits are limited by the corpus size and the specific compressor used.",
  "summary": "A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text by tokenizing it into subwords instead of raw bytes, why don't we tokenize text before handing it to a byte-level compressor like LZMA or zstd? It felt like the kind of idea someone must have already tried and quietly dropped. So instead of writing a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}