Building Roshni: A Real-Time, Multi-Agent Financial Voice AI for Bharat 🇮🇳

In India, financial inclusion has seen rapid growth fueled by digital initiatives like UPI, digital banking, and government-backed credit programs. However, many citizens, particularly in rural and tier-2/3 areas, face challenges when navigating complex financial terms, government schemes, and banking products. These hurdles often stem from cumbersome web interfaces and the need for immediate, clear spoken responses in their native language.

To address this issue, the developer created Roshni AI, a conversational financial assistant designed for seamless voice interactions in English, Hindi, and Hinglish. Roshni operates with ultra-low latency, ensuring a smooth user experience. The assistant's architecture includes real-time data pipelines that synchronize user audio input, speech-to-text conversion, LLM processing, text-to-speech synthesis, and audio output.

The stack employs Murf Falcon for Indian TTS voices (Anisha and Samar), Deepgram Nova-3 for multi-language STT, Google Gemini for the LLM engine, and LiveKit Agents for orchestration.

A key feature of Roshni is its ability to maintain persistent user memories through SQLite, enabling personalized interactions and tailored financial advice. Additionally, Roshni can seamlessly transfer calls to specialist agents (Roshni for general banking and Vikram for government schemes) using a multi-agent handoff mechanism. This architecture empowers Roshni to provide accurate, immediate, and culturally appropriate financial assistance to millions of users across Bharat.