AI bots are crashing university library sites and locking out the public
University libraries are being swarmed by AI bots, putting public access at risk
Data-scraping bots hunting content for LLMs are overwhelming academic library catalogues, sometimes knocking entire collections offline. One university logged more than 11,000 searches for Finnish music in a single day from thousands of IP addresses. Blocking IPs no longer works, CAPTCHAs also shut out legitimate crawlers, and requiring logins would put barriers on public access. Experts warn the extraction is discouraging institutions from digitizing at all.
Project Gutenberg is happy for its materials to be used as AI training data and even offers a one-click download of its entire library of books, but the bots are too stupid to use it.
- Bender
LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.
- blakesterz
What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.
- londons_explore
Behind nearly every AI bot is some human who will see real benefit from scraping my site.
I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.
If I put something on the internet for all to see, I want bots to be able to see it too.
I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.
- engineer_xyz
Same thing on the tiny scale. My side project site gets maybe 2 real visitors a month and a steady stream of 0-second "visits" from US datacenters. For a library with real users that load must be brutal.
- tokai
In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble.