BERT's tokenizer bottleneck was hiding before WordPiece

BERT's tokenizer bottleneck had been overlooked until the introduction of WordPiece normalization, which accounted for 88% of a warm English encoding pass in the bert-base-uncased model. The bottleneck lay in the WordPiece token selection algorithm, taking up only 8% of the time. Simon Brandeis detailed these findings in an October 15 Hugging Face article following CPU profiling and proposing an implementation.

The surprising revelation was the actual time spent by the tokenizer - not in finding token boundaries, but in preparing characters.

A tokenizer often appears to be a problem of identifying token boundaries, but the profile showed the cost was primarily in character preparation. This made tokenization worthy of separate measurement from the model in embedding or RAG services. However, it didn't establish tokenization as a dominant factor in any specific service.

The normalizer performed the same expensive work for regular characters, cleaning text, inserting spaces around Chinese characters, decomposing Unicode characters, stripping accents, and lowercasing the result.

The profiling exposed a costly check for nonspacing marks used by the normalizer to remove accents. It conducted Unicode property lookups even for ASCII letters that didn't require decomposition or accent removal. The profile highlighted repeated work on common cases rather than expensive rare inputs. The letter 's', for instance, went through a general Unicode path, decoding it, looking up its properties, and passing it through normalization.

Yet, its UTF-8 byte could immediately tell the implementation it was ASCII, allowing a cheaper path without altering its output.

The proposed branch introduced vector instructions to inspect 16 bytes at once, identifying UTF-8 bytes with a clear high bit to skip unnecessary Unicode normalization for ASCII. This shortcut benefited English and source code most in the reported corpora, while other scripts required different shortcuts based on script tendencies and nearby characters.

The implementation first checked if it could leave text alone, then avoided paying for unnecessary transformations. The branch achieved up to another 5.7 times faster BERT encoding, totaling around 50 times compared to tokenizers' 0.23 baseline.

These changes were still in a pull request when Brandeis published the findings. The speedup was measured in isolation, not as part of a complete RAG request, GPU inference, or application latency. The experiment's results showed that Chinese barely benefited, and some tiny inputs became slower. However, these findings offered a significant reason to examine normalization when tokenization appeared expensive, particularly for BERT-style embedding pipelines.