AI4Bharat to collect ten trillion tokens of data to power AI in Indian languages

Chennai-based AI4Bharat is collecting ten trillion tokens of language data from everyday conversations to technical documents across India's major languages. This data will power the next generation of artificial intelligence (AI) services, said Mitesh Khapra, cofounder, AI4Bharat.

Tokens are the basic building blocks that AI uses to understand language. They are usually parts of words or sometimes whole words.

“We have 200 million spoken words… four states where it is already live or in an active stage. We have use cases supporting farmers, children, digital payments and agriculture. Over the past three years, we have gone to almost every district in the country where we’ve tried to cover almost all the 22 official languages of the land,” Khapra said at the People+ai Mela in Bengaluru on Saturday.


AI4Bharat has ensured that it collects voice samples split across several demographics, across different professions, blue collar and white collar, he said, adding, “Several startups, academic institutes and deeptech institutes are using this data to build their own models to accelerate the adoption of language technologies.”

The tools required for data collection have been built from the ground up, according to Khapra. “Our data, models and scripts are open sourced. You can build on top of that,” he said.

Ten trillion token project

All the data collected over the past three years will feed into the Ten Trillion Token project, Khapra said. “This is going to be required to make sure that we are able to build native Indic models that support Indian languages and not as an afterthought. We want to collect ten trillion tokens in Indian languages that would be synthetic data that would be language information and cultural information,” he said.

To serve India’s diverse population, AI needs to understand Indian languages as well as it understands English, people+ai noted in its blog.

Building AI that works for India requires something different than what works in English. English data is everywhere on the internet, making it easy to train AI models.

India has 22 major languages, each with its own script, grammar rules and cultural context and the current AI approaches simply don't work well enough for this diversity, people+ai’s website said.

“When someone in Tamil Nadu asks an AI for help with a government form, or a farmer in Maharashtra needs crop advice, they should be able to do it in their own language, naturally and easily. But right now, that’s not possible. The AI models we have today stumble with Indian languages because they were built mainly for English,” people+ai said.

It further said, “That's why we started the ‘Ten Trillion Token’ project. We’re building the foundation for AI that can properly understand and work with Indian languages – from formal government documents to casual conversations at the local tea shop. Our goal is to collect and organise the massive amount of data needed to make AI work well for everyone in India, no matter what language they speak.”