Turn any website into clean Markdown chunks for RAG
Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then matches "Privacy policy | Contact us | © 2026" as often as it matches the answer. This post covers the two steps that fix most of it: converting each page to clean Markdown of its…
The article discusses two main steps to improve the quality of RAG (Retrieval-Augmented Generation) systems when using websites as data sources. The first step is converting each webpage's main content into clean Markdown format, while the second step involves chunking that Markdown based on its headings to create coherent sections.
The author, Aurelix, developed a tool called the Website to Markdown Actor to automate the Crawl and Markdown conversion process. This paid tool allows users to crawl within a specific scope, typically a section of a website like /docs/, rather than the entire domain. The Actor strips away unnecessary elements such as navigation menus, headers, footers, sidebars, cookie banners, and scripts, focusing only on the essential content.
The conversion process retains the structure of headings, lists, tables, and code blocks, which carry meaning in Markdown, while plain text would flatten these elements. The chunking follows a two-phase approach: first, splitting the Markdown by headings to ensure each chunk represents a single section, and second, splitting oversized sections at paragraph boundaries, but never within a code block.
Each chunk is annotated with metadata including the source URL, page title, and heading path (e.g., Page Section Subsection), which aids in retrieval and allows the model to cite its sources accurately.
The article emphasizes the importance of respecting the website owner's rules, such as following robots.txt guidelines and crawling at a moderate pace to avoid overloading the server. By applying these two steps—clean Markdown conversion and structured chunking—users can significantly enhance the quality of data fed into RAG systems, leading to more accurate and reliable retrievals.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.