The Numbers.com Collapse: How AI Bots and Prediction Markets Threaten the Web
What just happened to TheNumbers.com should worry us all
The definitive film data source The Numbers.com vanished after being overwhelmed by agentic AI bots and targeted by hackers seeking an edge in Polymarket prediction markets. Founder Bruce Nash explains how the site's thirty-year legacy infrastructure collapsed under the strain of automated scraping, forcing a rebuild. This incident reveals a fragile internet where AI lowers the barrier for cyberattacks, turning any data-rich website into a high-stakes financial target.
We now live in a world where a movie statistics website is worth hacking because prediction markets empower anyone to turn almost any data into money.
- primitivesuave
A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.
Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
- ethagnawl
At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.
Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.
- abetusk
I think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design.
The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:
> If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.
- podgietaru
"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products."
I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way.
I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anything (it absolutely wasn't) but because I had a problem, and I thought "heh wouldn't it be cool if someone else had a similar problem and could use my resource for it."
But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.
And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.
- djoldman
Just FYI, the bigger companies all allow you to block crawlers via robots.txt:
# Block Anthropic (Claude)
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
# Block OpenAI (ChatGPT)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
# Block Perplexity
User-agent: PerplexityBot
Disallow: /
# Block Google's AI Training
User-agent: Google-Extended
Disallow: /
User-agent: Google-Extended-Factual
Disallow: /
# Block Microsoft's Search & AI Crawler
User-agent: Bingbot
Disallow: /
- gajus
What a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.
- theragra
I have small website with archive of older radio broadcasts mostly in Russian. Lately, 90% of traffic comes from USA ;)
Previously it was like 5%.
URL is http://radar.lv btw
Thanks God i can serve up to a terabyte per month of traffic easily, otherwise it would be a catastrophe. I am not against bot scraping, but I worry about stability for meat visitors.
So, I think of enabling payments for website visits cloudflare recently developed.
I also have the problem of old technology like the mentioned site. While my personal blog uses static generator, archive website uses ancient Drupal version, which has no security patches for many years already.
- xeyownt
Prediction market as well as anything related to gambling must disappear from earth surface. Full stop.