10 Days to Build a Voice AI Tutor: The Good, The Bad, and The "Why Is It Silent?!"

In just ten days, a voice-based AI tutor known as Vidya Vani was created for the Indian market through the Murf AI's 10 Days of Voice Agents challenge. Vidya Vani is an intelligent AI tutor designed to help users practice spoken English and Mathematics with dynamic question generation, memory retention across sessions, live analytics, and seamless agent handoffs. The voice tutor is powered by Murf Falcon TTS and LiveKit WebRTC, both of which provide fast and efficient performance.

India, a country rich in diversity, faces a significant educational gap in terms of foundational education, specifically English literacy and Mathematics. While there are numerous ed-tech apps and text-based AI chatbots available, they often share a common issue when it comes to foundational learners: friction. Practicing spoken English with a text-based chatbot can be intimidating due to the need for spelling proficiency and typing speed, and these platforms do little to boost conversational confidence or pronunciation.

To address this problem, Vidya Vani was developed as a voice-first educational tool that eliminates the hassle of typing and screen-staring. Users can interact with the AI tutor by simply speaking to their phone or computer, resulting in a more natural, accessible, and human-like experience. Vidya Vani and her assistant, Aryabhata, were created as two distinct experts to focus on their respective subjects - Vidya Vani for English tutoring and Aryabhata for Mathematics.

The architecture behind Vidya Vani is complex, as it requires managing real-time, bi-directional audio streams. This was achieved using various technologies like LiveKit WebRTC for the transport layer, Deepgram for speech-to-text conversion, OpenAI/Anthropic LLMs as the brain, Murf Falcon for text-to-speech, and a local SQLite database for persistent memory.

The multi-agent orchestration system ensures that each expert focuses solely on their subject matter, preventing confusion and hallucinations that often arise from a single AI trying to address all subjects.

Some of the standout features of Vidya Vani include dynamic content generation, which produces unique exercises based on the user's needs, real-time analytics that allow teachers to track progress, and persistent memory that enables the AI to recall past interactions and greetings for returning learners. Although the development process presented numerous challenges, the final product represents a significant step forward in providing accessible, natural, and engaging educational experiences for users in India.