I Built a 500k-Domain Search Engine for Makers in a Weekend for $10
At 2am, annoyed by SEO sludge, I built a personal search engine for makers. Over a weekend, I crawled and summarized 560,000 homepages using a rented GPU and a local LLM, costing about $10. The system includes a fetcher, a worker that summarizes pages, a steward that blocks spam domains, and a search UI. Key challenges: 90% of the web is corporate, Tumblr and Neocities flooding, and GPU rental pitfalls. The final index is under 1GB, and the approach can scale to millions of sites for ~$100.
The headline, if you only read one paragraph: for about $10, an overnight GPU rental, and a few hours of steering the thing while it ran, you can have a personal search index of a few hundred thousand sites, under a gigabyte on disk.