Mana: 2-3 Seconds to Feeling Human

The story details the development of a voice AI assistant, named Mana, which processes user commands locally to provide responses within 2-3 seconds, as opposed to the typical 3-6 second delay caused by cloud-based voice assistants. The project began with the desire to create an AI that feels more human-like, drawing inspiration from characters like Alice in Sword Art Online, JARVIS, and Neuro-sama.

The primary challenge was to achieve this on a single machine with limited resources, specifically an 8GB VRAM GPU. Initially, the plan was to use multiple specialized models for different tasks such as transcription, planning, coding, verification, and text-to-speech (TTS). However, this approach resulted in significant latency due to model loading overhead and GPU memory constraints.

To overcome this, the developer opted for a single unified model, Qwen 4B, capable of reasoning, generating code, and explaining its actions all in one pass. This decision significantly reduced latency, with the final system averaging around 2-3 seconds before providing a response. This faster response time gives the user an impression of talking to a genuinely intelligent entity rather than a simple chatbot.

One of the key innovations in the system is the XML format used to separate the LLM's response into reasoning, code, and explanation. This structure allows for efficient processing and execution, where the code is executed silently, and only the explanation is spoken back to the user via TTS. This approach minimizes the delay caused by narration of code and ensures the user hears only relevant information.

The system also emphasizes VRAM efficiency, loading the 4B model only once (~2-3GB) and reusing it for each query, rather than loading multiple larger models repeatedly. Additionally, the TTS service is optimized by selecting appropriate quality modes based on whether latency is critical or not. The developer quantizes the models aggressively (using INT8 quantization) to balance accuracy and performance, acknowledging that a 5-10% accuracy loss is acceptable for voice queries.

The architecture of Mana consists of several components, including a local transcription model (Whisper), the main reasoning and code generation model (Qwen 4B), a TTS service (Kokoro/Chatterbox/Fish Speech), and additional features like a Live2D avatar for visual feedback. All these components operate locally on the user's machine, eliminating the need for external APIs and APIs latency.

The assistant maintains persistent memory, conversation history, and user preferences through an Obsidian vault, ensuring continuity across sessions. It also includes a local web search capability through a SearXNG instance, which conducts searches without relying on third-party tracking or API keys. This local search feature further enhances privacy and speed.

Mana is designed to run in the background without impacting system performance, even during gaming sessions. It can be activated by saying "hey Mana" and can execute tasks like summarizing screen content, performing code execution with sandboxing, and providing detailed explanations of its actions. The developer is currently working on integrating vision capabilities using a multimodal model and improving observability through structured logging.

Overall, the journey from the initial idea to a fully functional, local voice AI assistant highlights the trade-offs between performance, accuracy, and resource constraints. The result is a system that, despite its limitations, provides a responsive and engaging user experience that feels remarkably human-like.