Developer builds open-source search engine to surface the 'small web' for $10
Marlin uses small language models to create personalized search indices, bypassing corporate SEO to find independent creators.
Nottingham-based developer Alex Morley-Finch has released Marlin, an open-source tool that enables users to build their own personalized search indices. The project aims to reclaim the discovery of independent web content from the dominance of corporate search results.
Using a small language model to summarize homepages and a custom weighting system, Morley-Finch indexed approximately 560,000 pages. The entire operation required less than 1GB of disk storage and cost roughly $10 in cloud GPU processing fees. The technical architecture consists of four primary processes: a fetcher, a worker utilizing an OpenAI-compatible LLM, a steward for quality control, and an API paired with a web UI.
The fight against SEO
Modern search engines are frequently criticized for prioritizing corporate websites and content optimized for search engine algorithms (SEO), which often obscures independent creators. Morley-Finch developed Marlin specifically to target "the small web," focusing on portfolios, zines, and one-person software projects. By avoiding the storage of full-page HTML and focusing exclusively on homepage summaries, the tool prioritizes "people doing stuff" over commercial entities.
"I wanted a search engine of things I cared about, like 'portfolios, zines, weird little art projects, one-person software,' and other cases of just 'people doing stuff' that they share on the internet," Morley-Finch told The Register.
Challenging the infrastructure myth
The existence of Marlin challenges the prevailing industry assumption that web indexing requires massive, prohibitively expensive infrastructure. By demonstrating that a functional, specialized index can be built with minimal financial investment and storage, the project provides a blueprint for curated or weighted crawling. This shift allows individual users to regain control over the information they discover online rather than relying on opaque corporate algorithms.
Accessibility for hobbyists
Beyond the technical implementation, Morley-Finch suggests that the barrier to entry for creating such a tool is lower than previously thought. He noted that the project is cheap enough that the GPU bill should not deter others from attempting it, stating, "I think this is very doable in a weekend."
As the project remains open-source, the next phase for users will be determining how to refine their own weighting systems to surface specific niches of the internet. While the current proof-of-concept focuses on homepages, the scalability of this "small web" approach remains a key point of interest for those looking to bypass traditional search giants.