How I Built Memory for a Local AI Companion Without Sending Chats to a Server

Building a memory system for an AI companion without sending chats to a server presents unique challenges. The first version of memory was to simply save messages, which provided a history but lacked recall capabilities. As the chat history grew, sending it back to a local language model for every message became slow and contextually inefficient.

Thus, chat history and long-term memory were separated. Chat history provides immediate context for the model, while long-term memory consists of important facts, moments, preferences, and relationship details.

These memories are stored in SQLite, a local database, ensuring the app functions without a hosted database. Each character has their own data, so one character's memories do not inadvertently merge with another's. For each memory, the text itself, along with metadata like character association, creation time, importance, emotional weight, usage frequency, and vector embedding, is stored. The vector embedding enables semantic search, allowing the app to find related memories even if they don't share exact keywords.

When recalling memories, similarity alone isn't enough. Recent memories and those with higher emotional weight or previous usefulness should be given more consideration. The memory system extracts a small knowledge graph, representing relationships and entities, which aids in answering structural questions. However, the memory system must include an escape hatch for users to inspect or remove stored data, preventing the companion from feeling intrusive or "monitoring."

Ensuring the embedding model is downloaded correctly during setup prevents silent memory loss, a common issue that can undermine the user's experience.