It costs a lot more to use AI models in some languages
A new study from the Coalition for Language Equity in AI Research shows that AI models require significantly more tokens to process certain languages than others. These differences mean that users of underrepresented languages risk paying more for the same services, reports Rest of World . An LLM’s tokenizer breaks text down into blocks, or tokens , but it does so much more optimally for…
A recent study by the Coalition for Language Equity in AI Research reveals that AI models demand substantially more tokens to handle certain languages compared to others, potentially leading to higher costs for users of underrepresented languages. The reason lies in the tokenizer's efficiency, which performs better for languages with extensive digitized data.
Among the 20 languages with the most web presence, less common languages utilized 1.5 to 5 times more tokens than English prompts. This disparity escalates to 17 times in some models when examining 200 languages. Since most AI providers bill by tokens used, this leads to significantly higher expenses. Moreover, more tokens result in slower response times and a reduced contextual information capacity within AI models.
Researchers term this a "tokenizer tax," which intensifies digital inequality. This issue extends beyond languages with small speaker bases. For instance, Punjabi (~200 million speakers), Bengali (~266 million), Urdu (~288 million), Burmese (~36 million), and Amharic (~36 million) also experience this tax. Languages with uncommon writing systems, such as Burmese, Tibetan, Khmer, and Lao, are particularly impacted. In contrast, Latin alphabet languages often benefit, even when spoken by fewer people.
Written by urgent.news from Computerworld's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
