Stop Guessing Your RAG Hyperparameters: Why I Built a Local-First Benchmarking Framework

Developing Retrieval-Augmented Generation (RAG) applications from "Hello World" prototypes to production systems can take months. Tweaking hyperparameters like retrieval methods, rerankers, and Top-K values often feels like guesswork. Did the change improve the output or just introduce new edge cases? To measure these impacts systematically without costly cloud observability tools, I created Muffakir.

RAG development usually suffers from three key blind spots: limited visibility into the retrieved context and generated queries, chaotic experiment runs resulting in scattered logs, and difficulty reproducing results when document content changes. Muffakir addresses these issues with an open-source, local-first benchmarking framework.

The framework provides a ComposerUI dashboard to define search spaces and interactively tune retrievers, rerankers, Top-K limits, and prompts. Muffakir records granular execution traces for every trial, giving visibility into latency, cost, answer quality, retrieved context, and LLM-generated queries. It also ensures reproducibility with saved configurations and checkpoints. The architecture is designed for temporal benchmarking, allowing evaluation of how RAG systems respond to changing source documents over time.

To get started, install Muffakir via pip and launch the ComposerUI from the terminal:

pip install Muffakir[standard]

muffakir serve --open

If you build LLM applications and want to stop guessing about your RAG pipeline optimizations, try Muffakir. Check out the GitHub repository for feedback, feature requests, and contributions. What tools do you currently use to evaluate your RAG pipelines? Let me know in the comments!