Give Google the boot by building your own search engine

For those who are tired of search results overwhelming the web content they're genuinely interested in, building your own search engine could be the solution. Software developer Alex Morley-Finch from Nottingham, UK, created an open-source project called Marlin specifically designed for this purpose. Marlin catalogs around 560,000 homepages at a cost of roughly $10 by using a small portion of cloud GPU time and less than a gigabyte of disk storage.

Morley-Finch aimed to create a search engine centered around things he personally found interesting, such as portfolios, personal art projects, and software development by individuals who share their work online. His approach involved a crawler that only accessed homepages, a small local language model that read each homepage, summarized it into a name, two to three sentences, a category, and a handful of tags, and an API with a web UI for conducting searches.

The project comprises four main processes: a fetcher to collect domain names, a worker that interacts with a small language model similar to OpenAI, a steward that filters out undesirable pages, and an API with a web interface for tracking the process and making searches, including filters. Initially, the first version of Marlin worked smoothly, but after about a day, Morley-Finch found that the search engine had indexed the wrong content, primarily corporate sites and documentation, making up over 90 percent of the initial crawl.

To address this issue, he implemented a weighting system that prioritized specific pages in the crawl queue. After reaching his prioritized categories and starting to pull in irrelevant content, Morley-Finch stopped the crawl after gathering around 560,000 pages, spending approximately $10 on cloud GPU processing. He encountered a significant challenge in tagging and categorizing the indexed content, with the language model creating numerous category and tag names that were either too broad or applied infrequently.

This led Morley-Finch to develop a manual merge tool to clean up the resulting data, suggesting that his design may not scale well beyond a hobbyist project.

Despite these limitations, Morley-Finch believes that Marlin is feasible to build in a weekend and is inexpensive enough that the cloud GPU cost should not deter anyone from trying. He has made the code open source, allowing others to customize their own crawl based on personal interests and even rent a cloud GPU if needed. The Marlin GitHub repository includes detailed instructions and how-to guides for those interested in creating their own search engine tailored to their specific curiosity.