A web crawler for AI agents with no LLM in the crawl loop
When an AI agent needs an answer that lives a few clicks deep on a website, there are two common ways to get it. Dump everything. A crawler fetches many pages and the agent's LLM reads all of them. It is slow, it costs a lot of tokens, and the answer drowns in navigation menus. Let the LLM choose every link. This works, but each step is an LLM call. On my benchmark it cost about $0.19 per…
Jevcrawl is an open-source AI crawler that fetches answers from websites without relying on large language models (LLMs) during the crawling process. It offers a more cost-effective and faster alternative to traditional crawlers that use LLMs to read entire pages of content. By integrating a small scoring model to drive the crawl, Jevcrawl identifies relevant passages and returns them verbatim with their source URLs.
The crawler was developed to achieve the accuracy of the second option (LLM-driven crawling) at a price closer to the first option (page-based crawling).
Jevcrawl's workflow involves a scoring model, Jev, that answers yes/no questions with probabilities. For each page, the crawler sends two requests: one to determine if a passage is relevant to the question, and another to assess if a link leads to the answer. Relevant passages are kept, links are prioritized, and the process stops early if the kept passages already answer the question.
This approach reduces the number of tokens processed, resulting in a significant cost saving and faster response times. In a benchmark using WebWalkerQA, Jevcrawl fetched the answer page on 40% of tasks at a cost of $0.025 per task and completed the task in 27.5 seconds. In comparison, Claude picks each link option achieved 30% correctness at a higher cost of $0.194 per task and took 61.7 seconds. Other crawlers like Breadth-first crawl and Crawl4AI showed even lower performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
