Give Google the boot by building your own search engine

Nottingham-based software developer Alex Morley-Finch has created an open-source project called Marlin, which functions as a search engine tailored to the user's interests. The project utilizes a crawler that only indexes homepages, a small language model to summarize each webpage, and categorizes content into two to three sentences, a category, and tags. Morley-Finch did not incorporate any multi-tenant features, IP scanning, or full-page HTML storage, aiming for a simple, focused search engine.

Morley-Finch built Marlin using four main processes: a fetcher to gather domains, a worker process that communicates with a small, OpenAI-compatible language model, a steward to ensure only relevant pages are included in the index, and a web UI API for tracking the process and conducting searches. The initial development was completed within a few hours, with the system successfully summarizing content and enabling searches.

However, the first attempt resulted in indexing a broad array of websites, including corporate sites and documentation, rather than focusing on Morley-Finch's desired content.

To address this issue, Morley-Finch implemented a weighting system that prioritized specific pages and adjusted the crawl queue accordingly. Despite the system's initial success, Morley-Finch faced challenges with tagging and categorization. The language model assigned categories and tags freely, leading to a large number of distinct categories and tags, many of which were used infrequently.

He eventually developed a manual cleaning process to organize the data, acknowledging that the design may not scale well beyond a hobbyist project. Nonetheless, Morley-Finch believes that Marlin's complexity should not deter others from experimenting with building their own search engine, emphasizing that the project is open-source and affordable, even for those without advanced hardware.