Who Does Anubis Actually Stop? The Regressive Tax on Human Access
I built an AI tool to bypass Anubis, a proof-of-work system meant to block scrapers. While bots solve the challenge instantly, humans waste seconds and battery life on every visit. This creates a regressive tax that marginalizes users with weaker devices or non-JavaScript browsers, ironically failing to stop the very AI it targets while punishing real people.
The irony is that Anubis’s goal is to stop AI but it was incredibly easy for AI to circumvent it and yet the cost to humans and an open web remains.
- jdlshore
Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely.
Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors.
(That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)
- novafunc
Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas.
You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers.
It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to not be too annoying for humans, to not block real users, and not be privacy invasive.
- yellow_lead
> The exact adversary Anubis targets defeats it trivially.
Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive.
Anubis was not built to stop individual users with llms.
- nitwit005
This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping.
The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.
- throw0101d
I was curious about the name:
> Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots.
* https://anubis.techaro.lol/docs/
> The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice responsible for maintaining order in the universe. The heart was the seat of the life-spirit (ka). Hearts heavier than the feather of Ma'at were rejected and eaten by Ammit, the Devourer of Souls.
* https://en.wikipedia.org/wiki/Weighing_of_souls#Ancient_Egyp...