I Built a 500k-Domain Search Engine for Makers in a Weekend for $10
At 2am, annoyed by SEO sludge, I built a personal search engine for makers. Over a weekend, I crawled and summarized 560,000 homepages using a rented GPU and a local LLM, costing about $10. The system includes a fetcher, a worker that summarizes pages, a steward that blocks spam domains, and a search UI. Key challenges: 90% of the web is corporate, Tumblr and Neocities flooding, and GPU rental pitfalls. The final index is under 1GB, and the approach can scale to millions of sites for ~$100.
The headline, if you only read one paragraph: for about $10, an overnight GPU rental, and a few hours of steering the thing while it ran, you can have a personal search index of a few hundred thousand sites, under a gigabyte on disk.
- iFire
Here's my impressions of your algorithm:
1. read each site
2. rent a 4090 with https://vast.ai to run vllm
3. let llm model invent its own category and tag names freely
4. save 1KB of metadata each
a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags.
5. `code is going up as open source` soon (TM)
- frogger8
FYI for those needing a list of domains
Subject: I want all domains and subdomains
https://groups.google.com/g/common-crawl/c/XC2QmOE-sdI?pli=1
or google for COMMON CRAWL
- headz
TS;DR: Too Sloppy; Didn't Read.
- marginalia_nu
Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
- eichin
Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)