AI models cost more when you don’t use English

When AI models aren't designed to handle languages other than English, they can become significantly more expensive. A new study from the Coalition for Language Equity in AI Research reveals this hidden cost, known as the "tokenizer tax."

Tokens are the individual units of text or data that AI models process. The study found that some languages require far more tokens than others, particularly those with smaller online presences. Even within the top 20 most popular languages online, there's a noticeable gap in token usage. The most costly languages use 1.5 to 5 times more tokens than the least expensive ones. This disparity expands when considering 200 languages, with token costs increasing by up to 17 times.

This "tokenizer tax" doesn't just impact costs. Higher token counts lead to slower processing times and less effective context capacity. This is particularly problematic for regions with limited computing resources and connectivity. Inadequate language support also limits participation in the AI economy, making it harder for non-English speakers to fully engage in AI-based services.

Improving language processing involves enhancing the underlying data. Initiatives like RESPIN, which gathered text from 38 native speakers of different dialects, can help. However, the rarity of a language is not the only factor determining token usage. The script used for writing also plays a significant role. Languages like Asturian, which has few speakers but benefits from Latin-script advantages, fare better than languages written in less common scripts like Burmese, Tibetan, and Tifinagh.

The issue largely stems from tokenizers being primarily optimized for English and other high-resource languages. Companies developing conversational AI models are aware of this problem. However, simply improving training data isn't enough. Tokenizers need to be specifically designed to better handle diverse languages and scripts. Major AI labs should publish detailed cost analyses for different languages and commit to improving their support for underrepresented languages.