Building a Voice-First Learning Companion — My 10 Days of Voice Agents Journey

Over the last ten days, a reporter participated in the 10 Days of Voice Agents — VoiceForBharat Edition. The goal was to build a voice agent from scratch using Murf Falcon, a fast Text-to-Speech API. This story details what was created, the obstacles faced, and how others can construct a similar system.

The issue at hand concerns school-level students who require assistance with various learning and literacy subjects but often lack immediate assistance. Typing questions can also be a significant barrier, especially for younger students or those who are hesitant to use text. To address this, a voice-first learning companion was developed, allowing students to ask questions verbally, as they would to a teacher in person, rather than typing.

The voice agent is designed to assist students with learning and literacy topics across different subjects. For mathematics specifically, it hands the conversation over to a dedicated Maths Practice Specialist agent that can walk students through problems step by step. If the agent encounters a question it cannot confidently handle, it escalates to a human via email, ensuring students always have a reliable source of help.

The system operates by transmitting audio from the student through speech-to-text, into the main LLM agent, and back out through text-to-speech, all in real time. The main agent can also branch out to tools, a specialist agent, or human escalation when necessary. Key features include Indian voice powered by Murf Falcon, which provides a natural and familiar voice to Indian students; clear objectives and a defined escalation script; support for code-mixed language, enabling students to mix Hindi and English mid-conversation and receive responses in the same register; and a stateful frontend that displays the status of the conversation, such as Ready, Connecting, Listening, Speaking, and Call Ended.

The system also remembers returning students, greeting them by name and continuing where they left off, fetching exercises based on their level and scores, and initiating daily practice sessions at a time chosen by the student. In case of distress or the need for a teacher, the agent creates an escalation request and sends an email summary to a human, without revealing the full transcript.

Additionally, a call analytics dashboard tracks total, successful, and failed calls using real data without exposing any private information.

The challenges encountered included getting outbound calls to function properly. Since the agent needed to initiate contact without waiting for a user to connect, testing required a SIP client like Linphone, which presented some setup challenges. A more significant recurring problem was API token limits. To mitigate this, the reporter used API keys for multiple LLM providers, switching between them as needed when one ran low.

Fortunately, the switch was smooth with no compatibility issues. To build and run the project, clone the provided repository, follow the setup instructions in the README, add API keys to a .env.local file, run the project locally, and connect via the frontend to begin a conversation. The full code, including human escalation and specialist handoff logic, is available in the repository.

For future improvements, the reporter suggests reducing dependency on multiple LLM providers by optimizing token usage during testing, expanding outbound call testing to more real-world scenarios beyond Linphone, adding more subject-specific specialist agents beyond maths, and linking to additional resources. The repository for this project can be found at murf-livekit-starter.