Why India’s indigenous AI models need better data for regional languages

India's indigenous AI models require better data for regional languages, as they currently struggle to handle these languages effectively. The IndiaAI Mission recently announced three such models – EKA-1, Dhīr 2B and Dhīr 5B – that support 22, 10, and 10 Indian languages respectively. However, the models' performance hinges on the quality of data they are trained on, which is severely lacking for many regional languages.

"Low-resource" languages, such as Assamese, Bhojpuri, and Dogri, have very limited digitized linguistic data and computerized text in native languages. This scarcity hinders the development of robust AI models for these languages. Unlike high-resource languages like English, Hindi, and Arabic, low-resource languages have scant digitized text and speech data, making it challenging to train models on grammar, vocabulary, and sentence structure.

Data needed for AI models includes unlabelled text and speech, which helps in pre-training by familiarizing the model with grammar, vocabulary, and sentence structures. Nonetheless, unlabelled data alone is insufficient. It must be annotated with structural markers, tags, and metadata, a task that requires native speakers and domain experts. This process is labor-intensive and is not feasible for languages with scarce resources.

The lack of resources in low-resource languages stems from a lack of historical mass-produced literature and an uneven internet presence. The existing data in regional languages, when used for translation, often leads to culturally irrelevant content. Consequently, AI models trained on such data may suggest information that is not applicable in the local context, leading to misinformation in critical sectors like healthcare, education, farming, and local law.