Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA, in collaboration with IIT Madras and AI4Bharat, has introduced Nemotron-pretraining-Indic-v1, a massive Indic language pretraining dataset consisting of approximately 2.8 trillion tokens across 22 Indian languages. This release, accompanied by an open data generation pipeline, presents a substantial resource for model builders working on Indic pretraining.

The dataset includes native-script data for all 22 Indian languages and romanized text for twelve languages, representing a broad mix of curated English material, such as mathematics, code, and question-answer content. Each piece of data is traced back to a curated source document, and its output is checked against the original input, ensuring that the synthetic text remains grounded in real information.

The workflow behind the dataset comprises four stages: translation, quality assessment, repair, and script conversion. English documents are translated, quality checked, repaired, and converted into Latin script for the twelve dual-script languages. The remaining ten languages utilize a separate Indic Translate workflow. The dataset's accuracy and robustness are maintained through these quality checks and targeted repairs, ensuring the preservation of critical content such as equations, tables, and code.

By opening up both the data and the generation process, the team aims to make large-scale Indic data generation more accessible, enabling developers to contribute to stronger models across Indian languages. The dataset takes into account how people write in various languages, with native-script data and romanized text spanning 34 language-script configurations. The corpus is built using 15 Nemotron-CC-v2 subsets, which cover web text, question-answer material, mathematics, code, and other specialized content.