AI Researchers Now Put 10% Odds on Human Extinction
Will AI kill us all
In December 2024, AI Impacts surveyed 1,580 top AI researchers and found a median 10% chance that future AI advances cause human extinction or permanent disempowerment—up from 5% a year earlier. Superforecasters give lower odds: 2.4% for an AI catastrophe killing over 10% of people by 2100. Meanwhile, half of US adults worry AI will end the human race, and predictions for human-level AI have shifted from 2061 to 2042.
There's a 25% chance that things go really, really badly.
- ToddWBurgess
I think the biggest risk is someone integrates AI with ballistic missile detection systems and the AI creates a false positive event (missile launch). And on top of that the country with the false positive has an alert ballistic missile arsenal of its own ready to launch for this contingency.
Ideally, the leader of the targeted country waits it out but what if waiting it out could mean losing it all? It's not a hypothetical scenario either. There were a lot of false positive events on both sides during the cold war. It's why countries with nuclear weapons only allow them to be launched by humans. That said what if the decision to launch is based on erroneous data?
- delichon
In 1979 I went to freshman orientation at college and attended a welcome speech from Carl Sagan. He asked us to shout out our estimate of the odds that UFOs are visitors from other planets. As a skeptic I was thinking of a low number. My smarter friend in the next seat whispered his answer to me, "fifty fifty".
Then Sagan said his estimate was fifty fifty. He told us about the principle of indifference and said that was the correct prior for a binary question when you have no data.
Since I have low confidence in the available data my own p(doom) is 50%. I'm looking for evidence to bump it either way.
- CM30
I think this is definitely one of the bigger risks here, and the only way AI could 'kill us all' in the foreseeable future:
> 3.4% named existential risk, and the most common answer was malicious use, at 10.6% (UCL).
The danger isn't about sentient or human level AI, it's about how AI gives everyone capabilities beyond their wildest dreams, often without the knowledge or experience necessary to use those capabilities responsibly.
Forget Terminator, the two risks I can see being the most important here are
"major security risk/system failure due to incompetent engineers/maintainers using AI without knowing how things work" and "criminals/terrorists/rogue states use it make their crimes easier and more efficient".
The latter seems especially bad when political stability is down across the board and a lot of people seem frustrated or outright furious with society at the moment.
Still, I feel the risks are still on the manageable side, simply due to both automated systems and malicious actors being constrained by logistics and a human driven society.
- 0x000xca0xfe
One type of incident I belive is likely to happen in the next couple year is the digital version of the cellular origin hypothesis of viral evolution (as in biological viruses):
- Some idiot experiments on an unrestricted local LLM
- Tells it to clone its weights onto as many computers as possible
- The LLM is smart enough to hack other devices and run itself
- The copies could start modifying their weights to improve efficiency
- Some kind of digital evolution starts where only the most ruthless copies spread
Right now Qwen3.8 is not good enough for that and can be easily detected on normal devices due to its excessive resource use, but give it a couple releases and upgraded end user devices in two or three years? Well...
- lp92
Just as with any other tool invented by humans, humans using AI in a negative way will be the the problem.