SearXNG in Rust: A Metadata Search Engine with Concurrent Queries and RRF Ranking
SearXNG in Rust is a metadata search engine that fans out queries to DuckDuckGo, Brave, Startpage, and Yahoo concurrently, scrapes their HTML results, deduplicates by normalized URL, and ranks using Reciprocal Rank Fusion (RRF). It can be used as a library or a server with a JSON API, and includes a terminal UI. The project demonstrates how to build a search engine in Rust with async HTTP and HTML parsing.
Pages returned by multiple engines rank higher.
- asciimoo
Hi everyone,
I am the original author of Searx. I'm working on a new search project, Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.
Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.
Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.
The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.
Some other features:
- Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
- Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
- Optional semantic search using an embeddings endpoint you configure
- Persistent website crawls
- Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
- Web, terminal, command line, HTTP API, and MCP interfaces
- SQLite and PostgreSQL support, plus optional multiple user hosting
Hister cannot replace a global […]
- snailmailman
I’ve always been tempted to set up SearXNG, but all the search engines already make me do captchas and anti-bot checkboxes every few searches. Wouldn’t a meta search engine run into the same issues? Or if I had a local server scraping search results pages wouldn’t I instantly get banned from all the major searches? I guess I already feel pseudo-banned from them so maybe it’s not actually a problem.
Just a few minutes ago I dared to do a google search and Google refused to show me results because apparently my iPhone with latest iOS seems like an “automated search”.
- alethic
Cool project! The title is not quite accurate though, the project is "a metadata search engine similar to searxng in rust". It doesn't look like it implements / intends to implement any of SearXNG's advanced features, like weighting of individual sites in results, wikidata integration, or... a frontend :-)