Objective vs. Search: Decomposing What Makes a Good Tokeniser
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
