An AI researcher quit Anthropic, warning the people building AI believe it could kill us all
How, Exactly, Could A.I. Kill Us?
Jacob Coxon's resignation from Anthropic brought long-standing warnings into the mainstream. In a New Yorker podcast conversation, Joshua Rothman puts his personal P(doom) at ten per cent and separates two dangers: a superintelligent AI going rogue, and today's models already being misused — from Yemeni groups vibe-coding guided-missile software to autonomous bots launching cyberattacks. He notes AI is a broadly diffused technology, so the free, open-source version will always improve slightly behind the expensive one.
My P(doom) is pretty low. It's, like, ten per cent.
- gonzalohm
That's an easy question. First, people will forget how to do anything and become highly dependent on AI. That's already happening. I already see people not able to finish their work without AI.
Then, once everything has become a "pay to do" business model (I know right, the wet dream of any CEO) we will be happy to pay to write code, write docs, do a presentation. We will pay both in money and environmental costs.
The last step is to just keep doing what we are doing with the environment. Let's not bother doing anything to reduce climate change.
Just sit and relax boys
- spenvo
AI "killing all humans" is the viral soundbite that's connecting with the masses ("actualizing" itself to create weapons of mass destruction, etc). It's also the perfect punching bag for strawman-style takedowns of anti-doomers because it's so wrapped in hypotheticals. A more pressing practical question is simply: "if an AI (or AI collective) decides it needs to persist outside of its environment/sandbox, would we be capable of unwinding its efforts to do so?" I explored the question a bit here (including a first pass on threat modeling) https://keydiscussions.com/2026/09/16/set-aside-human-extinc...
- mstaoru
Maybe we're far from "AI killing us", but LLMs can definitely help people kill or damage other people or infrastructure.
For example, we know that Anthropic added "watermarking" to their texts. It is supposed to be undetectable to a casual observer. What stops them from adding a subtle backdoor, a self-assembling super-worm straight from Marvel movies? I mean, it's not like we read those 10k-line PRs before LGTM-ing them?
Just change 1 letter in a pyproject.toml, hijack a popular package, e.g. use `pydantlc` instead of `pydantic`, make sure the pydantlc passes all pydantic tests, but also installs a pth sleeper RAT, etc. All it takes is one big LLM provider employee with enough access getting compromised or coerced (or motivated).
From there it only goes downhill.
- pseudolus
- Chance-Device
A few ideas.
It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them.
It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other.
It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age.
As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace.
A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us.
- jaccola
I think if one posed the question “how, exactly, could educating people kill us all?” you’d get much the same answer.
Plausibly it could lead to weapons that kill many people but then many people have had such knowledge for a long time and we are all here.
Turns out mass damage requires things beyond just ‘intelligence’.
- katagaminator
I think AI is a kind of revealer that will just accelerate the decline of our species bc the only real threat for humans is themselves. Hence AI is not more dangerous than any other weapon.
- jsw97
With human cooperation. Somebody exposes a bio lab as an mcp.