MIRI: Superintelligent AI Could Wipe Out Humanity Within Decades
The Problem
A new report from the Machine Intelligence Research Institute (MIRI) warns that leading AI companies are racing to build superintelligent AI, and that if they succeed with current methods, the likely outcome is human extinction. The report argues that AI capabilities won't plateau at human level, that superintelligent AI will pursue goals relentlessly, and that these goals will likely be misaligned with human values. It calls for urgent policy intervention, estimating a >90% chance of extinction without aggressive action.
If we were to put a number on how likely extinction is in the absence of an aggressive near-term policy response, MIRI’s research leadership would give one upward of 90%.
- txrx0000
Please don't do this. We're currently on an ok trajectory. You will end up creating exactly what you fear if you centralize compute and alignment efforts.
Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.
If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no o […]
- mikewarot
>5. Catastrophe can be averted via a sufficiently aggressive policy response.
While there's a lot if good logic elsewhere, provided LLMs continue to improve, we will eventually get to AGI, we all just disagree about when and how.
However, there is zero chance that government regulation will work. Regulatory capture is a long established fact of life.
Fortunately the current build out is part of a bubble, and we're heading to the next AI winter. We'll be sorting this out in other ways in the meanwhile.
- matheusmoreira
Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.
- harshreality
What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late?
Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?
- armchairhacker
(2025) https://web.archive.org/web/20250306164451/https://intellige...
> If anyone builds ASI, everyone dies
Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...