Build Buddy, Day 5: my vector search's best match was a wrong answer

Day 5 of MLH Global Hack Week's Builders Week saw Build Buddy gather AI model explanations for C# compiler errors. Among the 12 real explanations - some correct, some wrong, and others inconsistent - were those from Gemma 3 4B and Grok 4.7. These were embedded locally and stored in a Tiger Data hypertable with a VECTOR(768) column and an HNSW index. When searching them like a RAG system, the vector search provided varied results.

When asking questions in plain English and ranking the explanations by cosine distance, the top match was Grok's explanation about the GetStringAsync method. It returned a Task and was correctly assigned as awaited, with a distance of 0.245. Another correct explanation about CS0029, about assigning a string to an int, was ranked 7th. However, the best match was wrong, as a statement missing its semicolon was the actual error. While a statement missing its semicolon was Gemma's explanation, it was incorrect.

Two filters were applied: trusted sources and a relevance threshold. Adding the WHERE review = correct filter removed the wrong answer, but as there was no correct semicolon explanation in the data, the search returned the closest leftovers. Even then, a threshold around 0.30 only caught unrelated answers, as the wrong semicolon answer scored 0.231, comfortably within that range.

Hence, retrieval needed both a trusted source and a relevance threshold. If neither is present, the system should state "I don't have a trusted answer."

The experiment used only 5 questions, one embedding model, and five explanations. The chosen 0.30 threshold was a demonstration rather than a benchmark, with full rankings, data, and scripts available in db/fix-notes-results.md. The Builders Week concluded on Thursday, with the next step being behavior checks to ensure fixes build and behave as expected, as just because an AI's answer looks right doesn't mean it is.