Anthropic's self-replicating AI worm: a threat that doesn't exist yet
Anthropic conjures threat that doesn't exist yet to stoke fears of AI apocalypse
Anthropic published research claiming self-replicating AI worms could soon spread across IoT devices, but the study's own details undermine the alarm: the exploit step was ignored, the test nodes were custom-built and deliberately left open, and only one model succeeded. The resulting worm would double in days, while Mirai doubled in an hour. It's another case of a frontier lab stoking fear of a threat that doesn't exist yet.
Yet again, a frontier lab is warning us of a threat that doesn't exist yet – at least in this case, basic security measures could make life harder for any future AI-powered botnet.
- overgard
This makes zero sense to me. Like, ok, so you can install a low powered model on low powered hardware and do inference extremely slowly... to what end? Why would that be a useful attack at all? Assuming you've compromised this hardware wouldn't it make vastly more sense just to install regular malware? And like, in what sense would an 8B or whatever model be of any danger anyway? I've used local LLMs and 8B
kind of sucks. Is it going to write bad poems on my TV or something?
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
- beloch
On the one hand, if they were really concerned about their software producing a self-replicating and adapting AI worm, they'd take steps to make sure that never happens. If it did happen, they'd be sued into oblivion. They probably just want to pump up their IPO and maybe get some regulatory lock-in to make life tough for late-starting competitors.
On the other hand, a self-replicating and adapting AI worm does sound legitimately pernicious. Not world ending, but possibly an end to the online world.
- cmiles8
The IPO failures last week are not depicting a market that’s keen for a $2T Anthropic.
In the bizzare world of today, rather than trying to convince folks that they have a viable business the AI labs appear fully dedicated to finding a story, any story, that lets them off the hook for tough business and financial questions they seemingly don’t have basic and good answers for.
“We’d be doing awesome financially right now, but we had to slow down because of the killer self replicating bots… humanity over profit!” is not a messaging that’s going to work in the public markets to explain quarter after quarter of massive losses.
- KevinMS
Please please regulate us, and don't forget all our competitors too, who cant afford to deal with it.
- 0x000xca0xfe
Did they just read my post from last week and take it as inspiration?
https://news.ycombinator.com/item?id=50002301#50006029
Local models are obviously not good enough (yet), you don't need "research" to know this.