Banning Anonymous Crawlers Won't Stop Bad Bots—It Will Cripple Journalism
Breaking the open web won't stop the bad bots
Over 300 media executives are pushing the Stealth Bot Prohibition Act, which would force every automated visitor to identify itself. But the bill misses the real harm: it doesn't limit request volume, compensate publishers, or restore lost referral traffic. Instead it threatens the anonymous scraping behind Pulitzer-winning investigations, the Internet Archive's trillion-page Wayback Machine, accessibility audits, and public health research. The authors argue liability should target actual harm, not anonymity itself.
Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful.
- thereader12
"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
- 09172307
This is exactly what companies like DataDome do, and sometimes Cloudflare continues to, when their bot protection pages enter endless loops. They don't even realize they are ruining the UX for thousands of legitimate users with their faulty JS browser fingerprinting verifications.
- udbhavs
Genuinely asking here: is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?
The article mentions the IETF is working on standards to deal with the proposed legislature and frustration of site operators. Almost any solution is going to be at the expense of user convenience and privacy. I see two interesting solutions mentioned here: micropayments, and proof-of-work. It seems unlikely the future of the open web will land on either. Both are inconvenient but at least could be done by anybody, potentially in a privacy preserving way. Using reputation based decentralized identity is another one (eg. proving you’re coming from a web domain or group), that seems more inconvenient for the platform because of false positives that could come through.
Meanwhile, a worse outcome is requiring attested hardware to get access to anything on the web. This already exists from the media companies pushing for controlled access to streaming. Netflix requires an hardware attested device to stream in 4K. Imagine a future where you now need one to read a New York Times article.
I can see some pushback against the alternatives (eg. digital ID), but if privacy preserving proofs are built in from the get go, it’s much better than not being able to see recipes because I have a rooted Android.
- armchairhacker
Micropayments, just enough to keep servers running.
I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.
Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.
IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.
- beloch
Scale seems to be the real issue. Given that there are relatively few actors with the scale to cause serious problems (and those actors have the resources to shrug off minor penalties), perhaps more targeted legislation is necessary. e.g. Different penalties for petty scraping and grand theft scraping. Of course, this would necessarily target the people who pay for election campaigns, so it will never happen.
- mlhpdx
I’m optimistic about NHP being the middle ground solution for a lot of things. With OS support it can be transparent to users yet still effective.
- bsammon
Sooo... the current status of this appears to be "referred to committee"
https://www.congress.gov/bill/119th-congress/house-bill/9915...
It's hard to tell (especially with my ignorance of current USHouse dynamics) if this is a real threat to pass, or if this is just a "Hey look, campaign contributor, I submitted this bill--look, I care about your issues" with no real intent/hope for it to pass.
As near as I can tell, the congress.gov website says it was introduced in late July, referred to committee the same day, and nothing has happened since.
I wonder, is there reason to think there's a recent push to move it out of committee that means we should actually start to pay attention to it?
- k__
I had the impression the bad boys are those that peddle an agenda on social media, not those who read a bunch of blogs.