Local RAG starts with retrieval, not infrastructure

RAG projects often begin by building infrastructure before they gather evidence. A database is set up, followed by a vector store, Redis, object storage, a parser service, a queue worker, and various dashboards. By the time the first PDF is imported, there are too many components, making a bad search result ambiguous. Was the parser incorrect?

Did chunking lose important context? Was the vector index empty? Did the query fail to match the document? Starting small is preferable. Use familiar documents, typical questions, and assess search results to determine their usefulness. An empty database is not a knowledge base. A knowledge base becomes valuable once it contains complex files: PDFs with tables, edited spreadsheets, product manuals with repeated headings, and contracts with specific numbers.

These files offer more insights than clean benchmark datasets. When testing retrieval, create short lists of questions with various failure modes: one with an exact name, number, or abbreviation; another that requires context rather than a single matching sentence; and one the document set cannot answer. A system that returns no useful results should indicate this clearly, otherwise people may blame the model for an answer the retrieval system never provided.

The system should also provide specific information about the source document, revision, and location of each result. Chinese queries often highlight retrieval weaknesses, as vector search performs well with semantic context but struggles with queries like product codes, contract numbers, names, or internal abbreviations. A hybrid retrieval system combining semantic and lexical paths can address both issues.

Local retrieval does not need to resemble production. An empty database, FTS5 with a vector extension, and Chinese tokenization can suffice for testing ideas. For larger deployments, consider PostgreSQL, pgvector, and a Chinese full-text extension. The components may change, but the core question remains: can this query locate the expected passage?

Local first does not mean local forever. Although SQLite with FTS5 and a vector extension can validate document processing and retrieval, it is not a replacement for a production database. Local mode is useful for testing document import, searching, resetting libraries, and repeating. Transitioning to production infrastructure is not overengineering; it addresses the need for a reliable, shared knowledge service.

After local retrieval works, connecting it to an agent is usually simpler than creating a chat interface around it. The knowledge service can offer search and document operations via an API or MCP, while the agent queries for relevant evidence when needed, avoiding the duplication of files across applications. For new RAG projects, start with a few messy documents and real questions. Only add more services once you can clearly articulate what each component accomplishes.