Stop Sending Raw HTML to LLMs
A scraper gives you a page as one HTML string: markup, inline styles, scripts, and somewhere inside it the text you wanted. Many pipelines send that string directly to an LLM. That works, so the cost rarely gets measured. But on the 10 pages I measured, the HTML used 3 to 24 times more tokens than the same pages as Markdown. A product page arrived as 267,361 tokens, of which 3,393 were visible…
Sending raw HTML strings to language models (LLMs) can be inefficient and costly. HTML includes markup, inline styles, scripts, and other elements, which contribute significantly to the number of tokens used. A comparison of 10 web pages revealed that HTML used 3 to 24 times more tokens than the same pages in Markdown format. For example, a product page arrived as 267,361 tokens, with only 3,393 being visible text. The guide explains where those tokens go, the cost involved, and how to avoid sending them.
Visible text constitutes only 0.3% to 18.5% of the tokens when HTML is used, while the remaining 81.5% to 99.7% is made up of markup. This surplus of tokens translates to increased costs, latency, and potential issues with rate limits. The model's response time increases when processing raw HTML, up to 5.7 times longer than when processing Markdown. Additionally, some models impose limits on the token size, causing rejections for large HTML pages.
Decodo's Web Scraping API can convert HTML to Markdown before sending it to an LLM, reducing the number of tokens significantly. For example, it returned 3,644 tokens for a Hacker News front page, compared to 11,817 tokens in raw HTML. The API keeps essential elements such as headings, lists, tables, and link URLs while removing unnecessary markup. Markdown conversion ensures that the model receives relevant information without the excess baggage of HTML.
When you need specific data like product prices, ratings, or review counts, HTML provides all the information, but Markdown extracts only the necessary data. In cases where ratings, stock status, or publication dates are missing from the Markdown output, an HTML parser can be used to retrieve the required information. In all tested scenarios, the LLM correctly answered questions based on both the raw HTML and Markdown versions.
However, sending raw HTML can lead to higher costs, slower response times, and potential rate-limit issues, highlighting the benefits of converting HTML to Markdown before processing with LLMs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
