How to Block Bots Without a CDN Using Simple Server Methods
How to Block Some of the Bots
I share nine practical methods to block poorly coded bots without relying on a CDN. These techniques range from enforcing HTTP/2.0 and blackholing data-center IP ranges to filtering suspicious user agents and client signals. While effective for hobby sites, I warn that these aggressive filters may block legitimate visitors like GoogleBot or VPN users, so extensive testing is required before deployment.
People always claim the user-agent is spoofed but in truth it rarely is.
- AussieWog93
I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
- fxtentacle
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
- userbinator
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
- CqtGLRGcukpy
If you are unable to read this, there is an archived copy at https://archive.ph/d3236
- binaryturtle
410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!
Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
- boznz
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
- genodethrowaway
and all valid traffic too, judging by these HN comments (and my own attempts to connect).
- inigyou
I can't read this article because I'm a bot, apparently.