Why I filed a consumer complaint against 7 global AI giants over the ‘linguistic tax’
When building production-grade enterprise artificial intelligence solutions in India, developers quickly confront an invisible financial penalty. Two users can submit the exact same prompt, identical in meaning and intent, to a commercial Large Language Model (LLM) API, yet one query will cost up to three times more to process than the other. The sole difference […] The post Why I filed a…
In India, developers constructing production-grade AI solutions encounter an obscured financial burden when utilizing commercial Large Language Model (LLM) APIs. Two users inputting identical prompts to these APIs experience varying processing costs based on the language employed. Founder of CogniTensor Technology, I experienced this uneven pricing firsthand when deploying AI systems for Indian-language workloads.
Recognizing the disparity affecting India's 600 million-plus non-English internet users, I recently submitted a formal consumer complaint to India's Central Consumer Protection Authority (CCPA) under the Consumer Protection Act, 2019, and the Legal Metrology Act, 2009, targeting seven prominent global AI API providers.
My methodology involved an offline audit designed to quantify the token disparity across current production models. Utilizing open-source tokenization libraries like OpenAI's tiktoken and Anthropic's Claude BPE tokeniser, I evaluated a standardized 10-sentence corpus across six languages: English, Hindi, Bengali, Tamil, Telugu, and Marathi.
The results revealed that English consumes 9 mean tokens per sentence while Hindi, Marathi, Bengali, and Telugu consume 21, ~1.70, 18, and 2 mean tokens respectively, with up to a 2.29x multiplier for Tamil. This linguistic disparity is a result of Byte-Pair Encoding (BPE) tokenisers trained predominantly on English and Latin-script web corpora, which compress common English words into single tokens, while Indic scripts are underrepresented in pre-training corpora.
At inference time, unmerged Indic words fragment into byte-level pieces, causing significant compute inefficiencies. This token measure is not standardized, leading to significant variance between different providers. This unequal treatment rises questions under India's Legal Metrology Act, 2009 and unfair trade practice provisions of the Consumer Protection Act, 2019.
Written by urgent.news from e27's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.