Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

Raw web pages are unsuitable for training machine learning models. Navigation bars, cookie banners, related articles, and advertisements clutter the content and dilute the signal. When fine-tuning a specialized model for legal reasoning, medical question-answering, or financial analysis, it is essential to have clean vertical text with provenance.

To address this issue, I developed an actor that transforms seed URLs into a token-aware JSONL dataset in a single run. The pipeline comprises four key stages: seed URLs to crawl (breadth-first search within the same domain), clean (removing boilerplate), deduplicate (exact and near duplicates), token-aware chunking, and finally, JSONL format with provenance information.

Two critical steps deserve special attention: Boilerplate strip, which eliminates script, style, navigation, footer, header, aside, and utility blocks (breadcrumbs, shares, comments, menus). Additionally, paragraphs where more than half of the links are anchor text are excluded, as they often indicate link farms rather than substantive content.

Deduplication employs exact hashing for identical records and a 6-gram Jaccard similarity threshold (default 0.95) for near duplicates, ensuring that syndicated copies of the same opinion are represented only once. Token-aware chunking maintains paragraph coherence, splitting at paragraph boundaries and hard-splitting on sentence or word boundaries when a single paragraph exceeds the token budget (default 512 tokens).

Every record contains metadata such as source (hostname), URL, domain (representing the vertical), title, and chunk index, enabling filtering, citation, and weight adjustments to the corpus later. Verification of the approach involved testing the method on three Supreme Court opinions from Cornell's Law School website (law.cornell.edu/supremecourt/text/...).

On a local machine, the crawler produced 559 chunks, each with a provenance key, ranging in size from 63 to 2,054 characters, with no duplicate texts. In a cloud environment, utilizing two seed URLs, the pipeline successfully generated 443 items, all containing non-empty text, tagged with the legal domain, sourced from law.cornell.edu, and free of duplicate texts.

The final dataset, stored in output.jsonl, comprised 443 lines (~811 KB) and a summary record detailing 443 chunks, 2 pages, approximately 173,000 estimated tokens, and no failed sources. It is crucial to note that seed URLs must serve server-rendered HTML; JavaScript-only single-page applications and websites protected by web application firewalls require a browser-rendering actor.

Chunk size is an approximation (around 4 characters per token), not an exact token count. Near-deduplication has a quadratic time complexity concerning chunk count, so large crawls should be capped using the maxChunksPerPage parameter. The crawler follows links within the same domain, ignoring cross-domain citations. To access the Vertical Corpora Builder tool, visit its Apify Store page.

For further reading, explore the author's other works, including building an MCP server on official registry data, scraping Romanian public contracts, creating a US building permits scraper, constructing an EU RAPEX product recall scraper, developing a water utility risk intelligence tool, and building an aviation hub API with data from multiple sources.