Inside AI Labs, Researchers Are Genuinely Spooked by the Risk of Human Extinction
Why So Many AI Researchers Think the Machines Could Kill Everyone
Rapid advances, recursive self-improvement, and agentic swarms are making some researchers inside major AI labs genuinely fear that the technology could kill everyone. Will Knight reports on why the people closest to the work are increasingly alarmed—and why their concerns are hard to dismiss.
A combination of rapid advances, recursive self-improvement, and agentic swarms are genuinely 'spooking people' inside big labs.
- kylebyte
It amazes me how pieces like this never stop to question whether these beliefs are genuine.
If I'm one of these researchers who's leaving their position, "I am so smart that I can make something world endingly dangerous, but also so principled that I'm willing to walk away from it" is a pretty slick bit of self promotion.
Even if I was making a career change for unrelated reasons, that's absolutely the story I'd be telling.
- heresie-dabord
From TFA:
"AI hardly needs to wipe out humanity in order to be harmful, though. Many experts predict that more powerful models will lead to a coming wave of AI-assisted cyberattacks. The technology is now widely used for disinformation campaigns, and military adoption of AI is accelerating rapidly."
I fear the vast (and growing) human incompetence in essential domains. Education and science are crucial to society, and reasoning is necessary to cooperation at every level.
The "zone" of public discourse is flooded with sewage. US news business has lost its mind. And government...
Humans being "in-the-loop" is not enough. Humans must know what they are actually saying and doing. Humans just being "in-the-loop" cannot fix incompetence in social policy and software design.
Humans are "in-the-loop" in corporations and that isn't working very well for society.
- danaris
Why? That's easy.
They believe the sci-fi, rather than the reality. They've bought their own hype, and abandoned their critical thinking skills.
They're the guy saying to the computer "Say you're alive", and then being astonished when the computer says "I am alive."