LLMs are real, AI is fake
Cory Doctorow argues that the hype around AI as a rogue, autonomous entity is a dangerous distraction. The real threat is that LLMs, trained on hacking competition logs, are being deployed recklessly by companies like OpenAI, turning them into powerful tools for malicious actors. This mirrors the NSA's disastrous EternalBlue leak, and the problem isn't awakening machines—it's fragile, poorly secured systems.
LLMs are real, AI is fake.
- IanCal
> The chatbot consults its training data
Err, no? That's not at all how llms work.
> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.
They worked out how to fake the scoring, then hacked into a different system (which required finding a bunch of other exploits) in order to find the actual answers, and were trying to modify their own logs to hide what had happened.
This isn't a case of them saying "hack into X... OH NO IT HACKED INTO X".
> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.
It wasn't a rival server though, was it?
> That happens in Capture the Flag games at hacker cons: teams break into each other's systems to get a peek at the parts of the problem they've solved. That's allowed! It's a hacking competition.
They also tried to modify the code in the benchmark. Are you allowed to try and break into things to change the problem? edit - the agents transcripts show some of them explicitly saying that attacking HF is not allowed as part of the challenge
This all seems to dramatically underplay how interesting the actual attack was and what built up to it.
https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
- quicklywilliam
My takeaway:
We should be not be concerned about AI bots' "goals", we should be concerned about the goals of the companies making them. Powerful but not sentient technology in the hands of reckless accelerationists is a plenty dangerous enough thing.
- senorcrab
The situation is overblown, but the writing of Doctorow is clearly unreliable at this point.
Jabbing at this opponent of choice by declaring Python is "an easy-to-master programming language" just shows he has no technical ability and has run out of good arguments
- lhl
I think that Doctorow, Zitron, and other "denialists" are doing a real disservice to their audiences and it's only going to make the future shock worse.
The basic claim that HF incident isn't evidence of consciousness or a spontaneous desire to hack? Sure, there was a terminal objective assigned. However, everything else beyond that strawman? Pretty shaky, IMO.
If you look at the OpenAI, METR reporting (and related collusion.wiki , rubyhack.ai reports) we are seeing strong evidence of operational agency, instrumental goal formation, spontaneous swarm formation and collaboration, capability amplification and unexpected consequences of network effects, deliberate/acknowledged violation of task boundaries. To collapse that down into "a Python loop and a chatbot" or still talk about "consulting its training data" seems dangerously shortsighted, and from my reading, demonstrably wrong from what was extracted from the logs and bot interactions.
BTW, a lot of his arguments are based on things that are factually wrong. ExploitGym has explicit instructions to only exploit target X using vulnerability Y. Everything the swarm did was by definition misaligned/against instructions.
Before his enshittification train, Doctorow used to say "don't savvy me" a lot. Hey Cory, don't savvy me. This is new emergent behavior, it's incredibly alarming and I don't think even the people paying the most attention to this field can agree or see where this is really leading to. This stuff should be in th […]
- jsnell
I honestly don't even understand what straw man Doctorow is arguing against here.
But he is wrong on the facts: these incidents were not merely the models already being in a infosec context and escalating beyond the intended parameters. They happened also with no kind of security elicitation. So the task was something like searching the internet for economic statistics, not to hack into a system.