Dario Amodei Wants an Antitrust Waiver and Open-Weight Regulation—Here's Why That's Dangerous

It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence.

We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.

So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.

>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.

As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.

While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down.

To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.

Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.

Maybe saying, "let's slow down", is another way of saying, "I love you."

Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."

Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"

If we can get everyone to slow down we can hemorrhage less money going into our ipo.

Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz.

Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.

AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.

I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.

Yes lets not control Open Weights etc.

But come one don't repeat stuff like this:

"Remember this man has been saying software development will be solved in “6-12 months” forever now."

Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.

These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)

I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.