{
  "id": 18439783,
  "title": "Why my 0.8 cosine threshold failed in an offline AI memory",
  "url": "https://urgent.news/2026/11/06/why-my-0-8-cosine-threshold-failed-in-an-offline-ai-memory",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-06T04:37:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/neox7/why-my-08-cosine-threshold-failed-in-an-offline-ai-memory-3nf2"
  },
  "original_language": "en",
  "account": "A self-taught robotics engineering student in Bengaluru created an offline AI assistant named Neo that remembers facts about users between sessions. Neo utilizes llama3.1 through Ollama, stores vectors in LanceDB, and facts in SQLite, with web search conducted via a self-hosted SearXNG. The assistant relies on a single rule: if the cosine similarity between a new fact and an existing one exceeds 0.8, the new fact is considered a contradiction.\n\nTo test the 0.8 threshold, the student measured 10 real sentence pairs using their embedding model through Ollama. However, the 0.8 rule proved ineffective for paraphrases and updates. Unrelated pairs only scored between 0.29 and 0.41, indicating that the useful range for the threshold was lower than initially assumed. Additionally, the student discovered an earlier error: the retrieval evaluation was measuring L2 distance instead of cosine similarity. After correcting the metric, Recall@k reached 0.95 and MRR reached 0.925.\n\nThe solution implemented involved abandoning the cosine similarity decision-making process entirely. Instead, the system identified potential candidates using a low threshold of approximately 0.5. If no candidates were found, a new fact was added. Once candidates were found, the flow moved to the LLM, which returned an action (ADD, UPDATE, DELETE, or NONE) and a target ID. The code then validated the answer, with the model proposing and the code deciding what was allowed. The LLM would select one of four actions, and the code would handle validation accordingly.\n\nThis approach proved effective in handling contradictory facts, as llama3.1 occasionally returned incorrect actions or selected the wrong target fact. Additional prompt tweaks further enhanced reliability by automating the decision-making process. The student advises others to test their chosen threshold on real sentence pairs before implementation, verify the metric used in their evaluation (L2 vs cosine), maintain a narrow LLM output for validation in code, and write tests with simulated LLM calls to check gate logic without relying on Ollama. The student also notes that v1.9 was largely written with Claude assistance and turned out unstable, leading to a manual rebuild of v1.9.5 from v1.8.5 for better understanding. The README includes credits to AI help, and KNOWN_ISSUES.md lists limitations such as the inability to extract comparative preferences and the loss of context in follow-up web queries. The assistant requires hardware capable of running an 8B model locally. The source code is available at https://github.com/Neo-X7/neo-AI.",
  "summary": "I'm a self-taught robotics engineering student in Bengaluru. I built Neo, a fully offline AI assistant that remembers facts about you across sessions. It runs llama3.1 through Ollama, stores vectors in LanceDB and facts in SQLite, and does web search through a self-hosted SearXNG. No cloud, no API keys. The problem When you tell the assistant \"I stopped liking chess\", it has to decide what that…",
  "key_points": [
    "Student created offline AI assistant Neo using llama3.1 and LanceDB",
    "0.8 cosine similarity threshold ineffective for paraphrases and updates",
    "Corrected metric improved Recall@k to 0.95 and MRR to 0.925"
  ],
  "editors_take": "Shifting from rigid mathematical thresholds to LLM-driven decision logic allows offline AI assistants to better navigate the nuances of paraphrasing and contradictory information during long-term memory management.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}