{
  "id": 9641468,
  "title": "Your RAG Pipeline Can Be Correct and Still Miss the Right Answer",
  "url": "https://urgent.news/2026/09/24/your-rag-pipeline-can-be-correct-and-still-miss-the-right-answer",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-24T20:04:20.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/your-rag-pipeline-can-be-correct-and-still-miss-the-right-answer?source=rss"
  },
  "original_language": "en",
  "account": "Your retrieval augmented generation (RAG) pipeline can appear accurate but still fail to provide the correct answer. Previously, Part 1 demonstrated a naive pipeline that incorrectly cited outdated policies. Part 2 addressed this by enhancing ingestion with structure-aware chunking, provenance on each chunk, explicit supersession, and a WHERE clause instead of a more sophisticated model. While retrieval is now accurate, the pipeline still has two issues: it is lossy and cannot answer certain types of questions. Neither problem can be resolved through chunking. Here's how to fix these issues. Understanding HNSW The behavior most people overlook significantly affects the pipeline's performance. ORDER BY embedding = query LIMIT 5 returns chunks that are probably among the nearest, not the actual nearest ones. HNSW (Hierarchical Navigable Small World) is an approximate nearest neighbor index that balances speed and exactness. The structure consists of a layered graph with sparse nodes in the top layer and denser nodes in subsequent layers. The search starts at the top layer, greedily moves toward the query vector, drops layers, and repeats. Three parameters control HNSW: m (default 16) represents links per node, higher values improve recall but increase index size; ef_construction (default 64) denotes the builder's effort to find good links, a one-time cost; and ef_search (default 40) determines the number of candidates kept during runtime. Adjusting ef_search impacts recall and latency. The default setting is optimized for prototypes, but it needs adjustment for production use. A common mistake occurs when applying filters. The original query in Part 2 looked like this: SELECT content FROM chunks WHERE status = current ORDER BY embedding = %s::vector LIMIT 5. However, with approximate indexes, the filter is applied after the index scan, not before. This means HNSW collects 40 candidates based solely on vector distance and discards them afterward. If 10% of your table contains superseded chunks, you'll end up with 36 survivors, averaging only 4 results from the 40-candidate list. Narrow filters, such as filtering by department, date range, or document, can lead to no results. The issue manifests as a successful query returning an empty set, which is more subtle than the Part 1 bug. The fix involves using pgvector 0.8.0's iterative index scans. Instead of scanning once and filtering, the index fetches candidates until enough survive the filter. SET hnsw.iterative_scan = relaxed_order; SET hnsw.ef_search = 100; relaxed_order provides approximate distance ordering, which is usually sufficient for RAG, while strict_order guarantees exact ordering, albeit at a slight performance cost. Additionally, set a maximum scan tuple limit (SET hnsw.max_scan_tuples = 20000) to prevent runaway scans on highly selective filters. Another important step is adding a normal B-tree index on the columns used for filtering. The vector index does not aid the filter, so a separate index on the filtered columns is necessary. CREATE INDEX ON chunks (status) WHERE status = current; This partial index addresses the original issue, improving recall and reducing scan time. When the filter remains stable, building the vector index as a partial index offers an even cleaner solution. CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops) WHERE status = current; While this approach is more rigid, it ensures exact results for the filter and reduces index size and scan time. However, it requires a new index for alternative filters and a rebuild if the filter changes. Despite this, it represents the cleanest available solution. One limitation of vector search is its inability to handle exact match queries effectively. For example, if the input is ERR_TXN_4412, the system returns passages about transaction errors but does not contain the exact string. This behavior is correct, as embeddings encode meaning rather than exact content. Exact match queries are better served by semantic similarity tools like a thesaurus, which are not suitable for finding specific identifiers such as error codes, SKUs, config keys, function names, ticket IDs, table names, version strings, etc. To address this limitation, it is recommended to combine lexical search with vector search and fuse the results. Postgres provides native support for lexical search, so no additional infrastructure is needed. ALTER TABLE chunks ADD COLUMN title TEXT; This column can store exact content, such as error codes or SKUs, allowing for accurate retrieval. By implementing these fixes, your RAG pipeline can achieve higher accuracy and provide more relevant results.",
  "summary": "pgvector, HNSW and the filtered-search trap: why your WHERE clause returns fewer rows than it should, and how hybrid search fixes what embeddings cannot.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}