I've Built RAG Infrastructure Several Times. Last Week Was the First Time I Actually Benchmarked It.

I have a confession: I've built RAG infrastructure multiple times before, yet I never benchmarked it until last week. Unit tests, integration tests, everything appeared green, and pipelines were connected. However, when asked if the retrieval was actually good, I shrugged and said, "Sure, deployment attached."

To address this, I started auditing the retrieval decisions I had shipped. Chunking contracts had undergone three revisions, and the fusion of vector and keyword search used RRF. A reranker was placed on top. However, none of these decisions were backed by numbers. My approach was based on architectural taste, and taste doesn't fail loudly.

The expensive part of an evaluation isn't the harness; it's the labeled data. To label nothing, I deterministically sampled 200 real queries from MIRACL-zh, a Chinese Wikipedia corpus with human-annotated passage relevance, under the Apache-2.0 license. I ran four configurations: vector only, FTS only, hybrid, and hybrid + rerank. Metrics included recall@10, MRR@10, and nDCG@10 against fixed qrels. Determinism was crucial; the same fingerprint, model versions, and code must produce identical metrics across runs.

To validate the harness, I first ran a smoke test with a mock embedding. Scores closely matched the random baseline, indicating the harness wasn't biased. However, the harness also needed to spin up a real standalone instance, create a knowledge base, and write to it. Surprisingly, this path consistently returned a 500 error. The issue stemmed from a missing deferred check on SQLite, causing the forward foreign key to fail.

Next, I discovered that the vector search had never worked in the production binary. The vec extension was missing from the build. This oversight went unnoticed by existing tests, as they mocked the database and ran against Postgres. When the real model was introduced, the results table displayed a row of 0.0000s, indicating zero recall in the FTS channel.

The hybrid search, however, scored 0.9799, identical to the vector-only configuration. This discovery revealed that my hybrid search had been running as a plain vector search with additional steps, unnoticed for an extended period.