Gentoo Bugzilla Shut Down After AI Scraper Overload
Gentoo bugzilla closed due AI bot scraper overload
Gentoo developer Michał Górny took Bugzilla offline, citing unusable performance due to LLM scrapers using thousands of IP addresses. He declined suggestions, stating he's not a sysadmin. Others discussed mitigation, including login-only access and blocking entire IP ranges, but noted scrapers often mimic normal traffic, making detection difficult.
No point in feeding the LLM scrapers that are using thousands of different IPv4 addresses, with no obvious patterns I can see.
- mrweasel
While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
- userbinator
Manufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?
- FranOntanaya
After a decade and half we had to restrict our public side and reorganize our old TED content because scrappers were really hungry for assorted captioned videos. If you have anything of value for training you get eaten alive if you stick out, it's like wearing short pants in the summer tundra.
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
- littlecranky67
It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
- eYrKEC2
I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
- ComputerPerson
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
- capitainenemo
Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
- rufasterisco
Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws.
This is not a tech problem. This is about what should or should not be legal.
Nor it’s a question of having time to implement solution X or Y.
If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop.
Ddos is already illegal.
I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.