The Escalating War Against AI Scrapers and Residential Proxy Networks
An Update on the scraper situation
The scourge of AI scrapers has intensified, with attackers using compromised residential devices to flood websites with traffic. While some operators are purely criminal, others like Bright Data disguise their networks as legitimate services. We are fighting back by optimizing our infrastructure and avoiding intrusive CAPTCHAs, but the arms race continues as shadowy entities race to harvest data for their own models.
These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.
- harshreality
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use ma […]
- mips_avatar
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
- georgyo
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
- sixtyj
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
- dang
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
- tingletech
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
- everfrustrated
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
- Bratmon
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
- eduction
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
- phendrenad2
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.