Anthropic and OpenAI Insiders Warn AI Could Kill Us All. An Engineer Says the Superintelligence Case Doesn't Add Up
The AI Superintelligence Debate Has Serious Flaws

Anthropic's Jacob Coxon resigned warning that his employer and OpenAI were racing toward self-improving superintelligence, and CEO Dario Amodei largely agreed. A Georgetown doctoral seminar examined whether recursive self-improvement is even feasible. The objections: RSI rests on unproven assumptions, diminishing returns and scarce training data may stall progress, Turing proved most problems are uncomputable, and alignment may be easier than doomsayers claim with the right system design.
Extraordinary claims require extraordinary evidence—and the idea of recursive self-improvement leading to an intelligence explosion is highly speculative, built on many unproven assumptions.
- saagarjha
I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.
On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.
For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.
For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”
- alecst
I think this article gets it wrong, which is unfortunate:
> Vernor Vinge pointed out, rapidity in cognition is not the same as intelligence. Speeding up the mind of a dog, he argued, wouldn’t enable the animal to play chess.
True, he writes about this in his essay "On the Coming Technological Singularity." His point is that there's some hypothetical possibility that computers will get fast but not smart--i.e. a dog thinking a million times faster than a human is still a dog. But seeing as computers have solved Jacobi Conecture, quasi-Riemann, and Navier Stokes--and we're in the early stages of LLMs--I think this hypothesis is bunk. The machines are smart. Foolish in some ways, but still smart.
> Their construction prevents such an outcome: Instead of prompting an LLM to generate an open-ended strategy for pursuing a general goal, these AIs consider much more limited spaces of possible actions
Well this is what Yudkowsky has advocated for--models that are trained on limited subspaces, e.g. protein folding sequences and nothing else. So I don't think this is a counterpoint.
On a personal note: Has "alignment" even been defined in a consistent way? How can AI even be aligned if people are not aligned?
> Objection #1: Recursive Self-Improvement Is Speculative
I don't think it's speculative since it seems to already be happening. OpenAI wrote about how use of agents in research is climbing quickly, and the goalposts for intelligence move every day ("Image generation can't do hands […]
- vanelsas
I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.
- mitthrowaway2
> Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible?
I get that the author is trying to frame themselves as a lone brave thinker, cutting through the noise with piercing clarity. But this question actually has received at least as much attention as the other points, even to the degree of inviting parody papers like "on the impossibility of supersized machines".[1]
- jeremyjh
I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.
- namr2000
I agree with the overall message of the article, but I do want to comment on:
> An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened.
Personally I would have expected solving Naiver-Stoakes, Quasi-Riemann, or Exponential Time hypotheses to require the same thing, and either:
* They did and AI swarms were capable of making such progress
* It was easier than we thought
Neither is particularly comforting. One could imagine that say continual learning techniques are similarly in reach and that could lead to something that looks vaguely like an intelligence explosion, even if it's not exactly the sci-fi horror that the boosters describe. It's really hard for me to model how likely that sort of thing is.
- jokoon
We don't even know what intelligence is.
I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.
If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.
What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?
AI can help for those things, but that's it.
We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.
- tescreal
It increasingly feels like anti-AI flavored arguments come in the form of moving goals further and further out. I don't think litigating an absolute frontier is useful overall, since it always ends up devolving into "intelligence of the gaps" where AI keeps filling gaps. There's always going to be some sort of ethereal special je ne sais quoi that humans possess more or less exclusively (much as a dog might have attributes that we cannot quite fully comprehend, even if we can describe it externally).
The claims meanwhile don't stand up:
* RSI is math and code heavy research (the things AI is improving most on)
* The allegation about large swaths of data being necessary is a common assumption. This one is really sticky: humans do not need to read the breadth and depth of a subject to comprehend a great portion of it; AI improvements should yield larger payoffs from same & smaller datasets
* Much learning and thinking is not about consulting existing work, but synthesis and recombination. AI has been revealing capabilities in this regard which should be factored in as it is further intensified (chain of thought being just the tip of the iceberg in my humble opinion)
I am curious to see if somebody would do a rigorous analysis of some of these suspicious underlying assumptions. It would help at least orient expectations better.