Give Google the boot by building your own search engine

If you're tired of search results drowning out the interesting bits of the web you actually care about, you could consider building your own search engine. Software developer Alex Morley-Finch from Nottingham, UK, created Marlin, an open-source project that catalogs around 560,000 homepages for roughly $10 using cloud GPU time and under a gigabyte of disk storage.

Morley-Finch built Marlin with a specific purpose in mind - to search for things he cares about, such as portfolios, zines, and one-person software projects that people share on the internet. He wanted a simple search engine with a crawler that only looks at homepages, a small local language model to read and summarize each page, and a basic categorization system with tags.

The project consists of four main processes: a fetcher to grab domains, a worker to interact with a small language model, a steward to filter out unwanted pages, and an API with a web interface for searching and filtering results. Morley-Finch built Marlin over a few hours initially, but later found that it had indexed corporate sites and documentation, rather than the specific content he wanted.

To improve the crawl process, he created a weighting system to prioritize certain pages over others. After exhausting his prioritized categories, the crawl reached around 560,000 pages and cost him approximately $10. Morley-Finch faced challenges with tagging and categorization, as the language model assigned many unique categories and tags, some of which appeared only once. He had to build a manual merge tool to clean up the results.

Despite these issues, Morley-Finch believes his project is feasible for a weekend project and is open-sourcing the code for others to create their own Marlin with a customized crawl based on their interests. The Marlin GitHub repository includes instructions and resources for building a personalized weighting list.