{
  "id": 4160898,
  "title": "Building a Hybrid RAG System with FAISS, BM25, and Agentic AI",
  "url": "https://urgent.news/2026/08/29/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-29T09:34:36.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/melvin_sabu/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai-h33"
  },
  "original_language": "en",
  "account": "Building a RAG System with FAISS, BM25, and Agentic AI\n\nIn my recent AI Engineering project, I created a Hybrid Retrieval-Augmented Generation (RAG) system that utilizes both FAISS vector search and BM25 keyword search to retrieve pertinent information from a knowledge base. This system then provides the retrieved context to a language model (LLM) to generate accurate and grounded responses. Here's a detailed breakdown of my project and the lessons I learned throughout the process.\n\nWhy RAG?\nLarge language models are proficient at generating natural language responses; however, they might not have access to information within a particular document or knowledge base. RAG tackles this limitation by initially retrieving relevant information from an external knowledge base and subsequently supplying that information as context to the LLM.\n\nThe basic workflow of this RAG system is as follows:\nUser Query → Retrieve Relevant Information → Provide Context to LLM → Generate Answer\n\nFor this project, I aimed to enhance this workflow by incorporating both semantic search and keyword search.\n\nHybrid Retrieval\nThe system employs two retrieval methods, namely Vector Search with FAISS (Facebook AI Similarity Search) and Keyword Search with BM25 (Best Matching 25).\n\nVector Search with FAISS:\n- Document content is divided into smaller chunks and converted into vector embeddings.\n- These embeddings are stored in a FAISS index to identify documents that are semantically similar to the user's query, even when the wording differs.\n\nKeyword Search with BM25:\n- BM25 focuses on the occurrence and relevance of terms in both the user's query and the documents.\n- It is particularly useful for exact terminology, technical terms, names, and identifiers.\n\nBoth retrieval methods are combined by normalizing and combining the FAISS and BM25 scores using weighted scoring. The highest-ranked chunks are then used as the final context for generating the response.\n\nDocument Chunking\nPrior to retrieval, documents are divided into smaller chunks. Chunk size is crucial as it directly impacts retrieval quality. Extremely small chunks might lose essential context, while excessively large chunks could contain unnecessary information, negatively affecting retrieval precision. Consequently, finding a balance between context and retrieval accuracy is essential.\n\nEach chunk is accompanied by metadata, such as its document ID and title, which simplifies identifying the source of the retrieved information.\n\nAdding Agentic RAG\nAnother aspect of the project involved integrating the retrieval system as a tool for an agent. A knowledge-base search tool was developed as follows:\nknowledge_base_search(query)\n\nThe agent can utilize this tool to retrieve relevant information before the LLM generates the final response. The workflow transforms into:\nUser Query → Agent → knowledge_base_search() → FAISS + BM25 → Hybrid Ranking → Retrieved Context → LLM → Final Answer\n\nThe agent is instructed to incorporate the retrieved information when responding to questions and should refrain from generating unsupported information. If the knowledge base lacks sufficient information, the system can indicate that the information is not available.\n\nLLM Choice\nFor response generation, I employed Qwen2.5-72B-Instruct through InferenceClientModel. The LLM does not perform the initial retrieval; instead, the retrieval tool provides pertinent information to the agent, which is then utilized as context during response generation. This separation between retrieval and generation ensures that the knowledge base remains the primary source of information.\n\nImplementation\nInitially developed in Google Colab, the project was later consolidated into a single Python application in VS Code. The main pipeline comprises:\n- Documents\n- Chunking\n- Embeddings\n- FAISS Index + BM25 Index\n- Hybrid Retrieval\n- Ranking\n- Agent Tool\n- Qwen2.5-72B-Instruct\n- Answer\n\nThe embedding model and retrieval indexes are initialized when the application starts, ensuring that the entire retrieval pipeline does not need to be rebuilt for every query.\n\nTesting\nI assessed the system using questions related to the information available in the knowledge base. For example:\n\"Why is document chunking important in a RAG system?\"\n\nThe hybrid retrieval system prioritized the \"Document Chunking\" source as it directly addressed the question. The retrieved context was subsequently forwarded to the agent and employed to generate the final response. I also considered questions where the required information was not present in the knowledge base. In such cases, the system was designed to avoid merely relying on the LLM's general knowledge.\n\nKey Takeaways\nWorking on this project allowed me to understand that building a RAG system requires more than just connecting an LLM to a vector database. Numerous components significantly impact the quality of the final answer, including:\n- Document chunking\n- Embedding quality\n- Retrieval strategy\n- Keyword matching\n- Hybrid ranking\n- Context construction\n- Agent behavior\n- LLM generation\n\nOne of the most significant takeaways from this project was recognizing the paramount importance of the retrieval stage. Even the most powerful LLM can produce a subpar response if the relevant information is not retrieved accurately.\n\nFuture Improvements\nSeveral enhancements can be explored for the current system, such as:\n- Implementing better chunking strategies\n- Conducting reranking of retrieved documents\n- Employing query rewriting\n- Enhancing semantic chunking\n- Improving relevance filtering\n- Automatic retrieval and answer evaluation\n- Automating the optimization of hybrid retrieval weights\n\nConclusion\nThis project granted me practical experience with Hybrid RAG, vector search, BM25, FAISS, embeddings, agentic tool use, and LLM-based generation. The essential insight I gained is that good RAG is not solely reliant on the LLM; the quality of the retrieved context holds equal importance. By combining semantic and keyword-based retrieval, the system can leverage both the meaning of a query and the specific terms within the documents, resulting in more accurate and contextually relevant responses.",
  "summary": "As part of my AI Engineering journey, I recently worked on a project that helped me understand how Retrieval-Augmented Generation (RAG) works in practice. I built a Hybrid RAG system that combines FAISS vector search and BM25 keyword search to retrieve relevant information from a knowledge base and use it to generate grounded answers. In this post, I’ll briefly share what I built, how the system…",
  "key_points": [
    "Hybrid RAG system combines FAISS vector search and BM25 keyword search",
    "Documents chunked before retrieval, balancing context and retrieval accuracy",
    "Agent tool utilizes FAISS + BM25 hybrid ranking for context construction"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}