I Built My Own "Muse Charm" on an ESP32 Stick: One Fast Brain, One Slow Brain

The story revolves around a push-to-talk AI device called Kai, which is built on an ESP32 Stick. Kai is a small, keychain-sized device that can engage in conversations with users, perform research, and schedule tasks. It operates using two brains: a fast path using Meta's Gemini Live model for quick spoken answers, and a slow path utilizing a Hermes Agent for more complex tasks like memory, research, and scheduling. The voice model itself decides which path to take, guided by an evaluation set to maintain accuracy.

Kai's hardware consists of an M5StickS3 development board, featuring an ESP32-S3 processor with Wi-Fi and Bluetooth capabilities, 8 MB of flash memory, and 8 MB of PSRAM for audio buffering. An ST7789 LCD display, an audio codec, and a speaker enable audio input and output. The device runs on a 250 mAh LiPo battery, with power management features to optimize energy consumption.

The architecture of Kai is designed to balance speed and efficiency. The fast path is powered by the main processor cores, capable of handling real-time tasks like speech recognition and generation. The slow path relies on the always-on RTC timer and RISC-V ultra-low-power coprocessor, allowing Kai to perform research, manage schedules, and handle background tasks without consuming significant power.

This dual-brain system enables Kai to effectively manage a wide range of tasks while minimizing power usage, making it suitable for a pocket-sized device.