Why my 0.8 cosine threshold failed in an offline AI memory

A self-taught robotics engineering student in Bengaluru created an offline AI assistant named Neo that remembers facts about users between sessions. Neo utilizes llama3.1 through Ollama, stores vectors in LanceDB, and facts in SQLite, with web search conducted via a self-hosted SearXNG. The assistant relies on a single rule: if the cosine similarity between a new fact and an existing one exceeds 0.8, the new fact is considered a contradiction.

To test the 0.8 threshold, the student measured 10 real sentence pairs using their embedding model through Ollama. However, the 0.8 rule proved ineffective for paraphrases and updates. Unrelated pairs only scored between 0.29 and 0.41, indicating that the useful range for the threshold was lower than initially assumed. Additionally, the student discovered an earlier error: the retrieval evaluation was measuring L2 distance instead of cosine similarity. After correcting the metric, Recall@k reached 0.95 and MRR reached 0.925.

The solution implemented involved abandoning the cosine similarity decision-making process entirely. Instead, the system identified potential candidates using a low threshold of approximately 0.5. If no candidates were found, a new fact was added. Once candidates were found, the flow moved to the LLM, which returned an action (ADD, UPDATE, DELETE, or NONE) and a target ID.

The code then validated the answer, with the model proposing and the code deciding what was allowed. The LLM would select one of four actions, and the code would handle validation accordingly.

This approach proved effective in handling contradictory facts, as llama3.1 occasionally returned incorrect actions or selected the wrong target fact. Additional prompt tweaks further enhanced reliability by automating the decision-making process. The student advises others to test their chosen threshold on real sentence pairs before implementation, verify the metric used in their evaluation (L2 vs cosine), maintain a narrow LLM output for validation in code, and write tests with simulated LLM calls to check gate logic without relying on Ollama.

The student also notes that v1.9 was largely written with Claude assistance and turned out unstable, leading to a manual rebuild of v1.9.5 from v1.8.5 for better understanding. The README includes credits to AI help, and KNOWN_ISSUES.md lists limitations such as the inability to extract comparative preferences and the loss of context in follow-up web queries.

The assistant requires hardware capable of running an 8B model locally. The source code is available at https://github.com/Neo-X7/neo-AI.