Running RAG with a (small) language model in the browser

This article explains how to run a small language model in the browser for Retrieval-Augmented Generation (RAG) without needing a server or API key. Using the MicroLLM lab as a reference, the author builds a system called "ask my blog" that finds the four most relevant passages from a blog post and summarizes them in two sentences using a small language model.

The passage retrieval process is similar to the homepage's static-embedding search, except that instead of aggregating results into posts, it retains the best four passages. If the best passage scores below a threshold, the model is not called, and the page informs the user that the question couldn't be answered based on the available data.

The chosen model must be reasonable to download for readers, hence the preference for a few hundred megabytes rather than gigabytes. The comparison includes models such as SmolLM2 (135M and 360M parameters), Gemma 3 270M, LFM2.5-350M, Qwen2.5-0.5B, and Qwen3-0.6B. Among these, Qwen3-0.6B is chosen due to its reasonable size of 579 MB, which fits within the 350 MB budget.

While models smaller than 350 MB don't meet the criteria, the author evaluates how faithful each model is to the provided passages, measuring both faithful answers and invented answers (answers with claims not present in the passages) for up to 30 questions.