The many futures of superintelligent AI, from utopia to extinction
Possible Futures

A taxonomy of possible futures for artificial superintelligence, organized around two questions: whether we reach ASI and whether we solve alignment. Scenarios range from a Butlerian Jihad banning thinking machines, to a mundane world where AI remains a complement to human labor, to a permanent underclass as automation eliminates jobs, to a singleton ASI imposing its values, to a MAD world of personal ASIs, and to a power concentration where a few companies divide the cosmos.
The future is a very long time: the stelliferous era will last at least 100 trillion years. Maybe the singleton is aligned at first, but over millions or billions of years of growth and self-modification, the alignment drifts.
- lnrd
In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? It hacks all nuclear launch sites and starts to make demands that we are forced to follow? Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Even so can't it be stopped by turning off datacenters and network connections? How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
I think it's too big of a question to leave it as an unexplained implication.
Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight?
Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard.
I find this article good food for thought, but it misses some crucial pieces.
- txrx0000
The current trajectory looks closest to the Orion's Arm scenario, though probably not exactly that because we may end up in symbiosis even without spreading out. The artificial is a subset of the natural rather than a completely separate category. This is probably how it has to be anyway - each and every one of us will take our own place in the grand ecosystem, as has always been done in the past. The universe, or at least this part of it, is still too young and volatile to settle into simple steady states.
- Animats
"Full automation" is somewhat independent of artificial general superhuman intelligence. Full automation requires good robots and a good solution to unstructured manipulation, but not unusually high intelligence. Near full automation may lead to the permanent underclass scenario.
Consider near full automation of Amazon. A shipping container is unloaded from a ship by a robotic crane, which places it on a robotic transport platform. The transport platform moves the container to a container stacking area, where another robotic crane puts it on the container stack. Later, a truck pulls up to a truck dock, and the stacking crane loads a container onto it. This is all standard procedure at the more advanced ports today.
Currently, there's a bit of human intervention at this point where a human has to set the twist-locks that lock the truck to the container. That's a good job for a humanoid robot. Then the truck drives off. The truck could be self-driving right now; that's being demoed. The truck drives to a fulfillment center and backs into the dock. Unloading robots and forklifts unload the truck. That's been demoed, but doesn't work well yet. The contents of the truck go onto conveyors.
Next comes something that hasn't been automated yet - opening boxes and separating the contents. That's an unstructured manipulation problem. There's slow progress in that area. Once separated, scanned, and put into totes or bins, the incoming products go into the robotic storage system, somethin […]
- pron
We live in a world where many people, even in advanced countries, reject 18th- and 19th-century science, let alone more modern science. I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]. Not to mention that many if not most systems in the world are non-linear, and so no amount of intelligence can predict and control them, certainly not without gathering a lot of resources. Even the question of "alignment", which is presented as a simple matter, is obviously not so simple, because humans are far from being aligned with each other [2], and their values sometimes differ (to the point of coercion or killing).
This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [3] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.
[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?
[2]: I don't think I'm "aligned" with Elon Musk or the US government, and they already have some power over me, not thanks to their intelligence but thanks to their control over resources.
[3]: Not only are the most intelligent people not the most […]