LLM scrapers drove this self-hoster to consider killing his photography site
Running web sites makes me not want to
A self-hoster describes how LLM scrapers overwhelmed his home server, saturating his upload bandwidth with generated content and forcing him to block large cloud IP ranges in nftables. Frustrated that his photography is being slurped up to feed corporate models that sell back a poor imitation of creativity, he's considering pulling his digital images entirely and selling physical zines per event instead.
It really irks me how entitled the corporate scraping of the web has become just so they can sell back to us a poor imitation of creativity.
- baumschubser
from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.
- eska
I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things"
Can anyone corroborate this?
- nightdeployboy
I also run a small side project site, and I get hundreds of thousands of bots visiting every week..
- yosefk
serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine
- Bender
Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble.
Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.
if ($server_protocol = HTTP/2.0) { set $limit_rate 384k; set $ok 1; }
if ($http3 != "") { set $ok 1; } # no rate limit for quic
if ($ok != 1) { set $limit_rate 32; limit_rate_after 128; set $trace "${trace}not-h2h3 "; add_header Content-Type text/html;return 200 '<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>body{width:35em;margin:0 auto;font-family:Tahoma,Verdana,Arial,sans-serif;}</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p></body></html>'; }
People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC.
This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or […]