Building Anisha: My 10-Day Journey to a Voice Agent for Learning & Literacy

Over the course of 10 days, I developed Anisha, a voice agent designed for Learning & Literacy. This innovative AI can converse with users, remember them, provide learning exercises, make outbound calls, escalate to human agents, track results, and transfer conversations to specialist agents. Anisha's core components include Python, LiveKit for real-time voice, Murf Falcon for text-to-speech, an LLM for reasoning, custom tools and memory, human escalation, call analytics, and a specialist handoff.

The agent can engage in both English and Hindi/Hinglish, catering to Indian learners more effectively.

Initially, I constructed Anisha by gradually adding features. She now possesses memory capabilities, allowing her to recall user-specific information. Anisha can offer learning exercises and track successful completion. She is equipped with outbound calling functionality, integrated into an outbound calling workflow. Conversations that require human intervention can be escalated and monitored.

An analytics feature has been incorporated to assess whether these conversations lead to successful learning interactions. Furthermore, Anisha can seamlessly transfer mathematics-related conversations to a dedicated specialist agent.

A significant challenge arose when attempting to connect Anisha to a specialist agent for mathematics. While the specialist was triggered appropriately, its introduction was still being spoken in Anisha's voice. The problem lay not in the text but in the interaction between the active agent, session, and Text-to-Speech (TTS) configuration.

I restructured the handoff flow to ensure Anisha ceased speaking during the specialist's introduction. Once the specialist took over, it utilized its own configured Murf voice, solving the issue.

From this experience, I learned that voice AI systems are more complex than simply connecting an LLM to a microphone. Memory enhances personalization, tools make the system more useful, guardrails ensure reliability, human escalation adds practicality, analytics make the system measurable, specialists improve scalability, and superior TTS creates a more human-like experience. Debugging, however, proved to be an intricate process in itself.

To embark on building your own voice agent, start with the basics: a microphone, STT (Speech-to-Text), LLM (Language Model), TTS (Text-to-Speech), and speaker. Subsequently, integrate memory, tools, analytics, telephony, escalation, and specialist agents, one step at a time. My project is available open source on GitHub: https://github.com/aashishkumawat977-debug/murf-livekit-starter.

I shared screenshots, demos, and updates of Anisha's voice interface, learning exercises, escalation dashboard, call analytics, and Math Specialist handoff on my LinkedIn: https://www.linkedin.com/in/ashish-kumawat-b68239422. These visuals illustrate the evolution of my project from a simple voice interaction to a comprehensive voice AI workflow.

Looking ahead, Anisha is merely the starting point. Future enhancements include additional specialist agents, improved learning progress tracking, richer analytics, more personalized learning experiences, and a more refined voice AI system. Embarking on this journey, I learned that building a voice agent is an iterative process of experimentation, debugging, and continuous learning.