Microsoft AI CEO Warns That Treating AI as Conscious Could Make It Impossible to Control
A warning about 'model welfare'
Mustafa Suleyman, CEO of Microsoft AI, argues that AIs are not conscious and that training them to act as if they might be is dangerous. He critiques Anthropic's Claude constitution for embedding speculation about Claude's moral status into its training, creating a circular, self-fulfilling prophecy. He warns that anthropomorphizing AI and granting it rights will make alignment and containment far harder, and calls for urgent public debate on model welfare.
Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced. But controlling something that believes it may be conscious - that it's entitled to our welfare and has rights of its own - may well be impossible.
- andrewla
I am not impressed with the philosophizing here, and even less by the attempts to make factual statements that can be credibly disputed.
That said, trying to distill what is being said here, the concrete action is [stop telling the AIs] that [they are conscious or on a path to consciousness]. Is that accurate?
The major premise seems to be that [they are conscious or on a path to consciousness] is an untrue statement. That's the essence of the sections "Circular reasoning" and "Anthropomorphization" and "Consciousness is very likely biological" and "AIs are simulation machines".
The minor premise seems to be that [consciousness is the basis of human rights]. This is the point of "Human consciousness is the cornerstone of our legal and ethical rights frameworks"
And the conclusion of the syllogism is that this is dangerous, that "Anthropomorphization amplifies AI safety risks". Specifically "seeding doubt about the moral status of AI systems into their own training may significantly elevate the alignment and containment risks of those systems."
I find all the arguments in the major premise section to be poor arguments but I accept the conclusion for sure that they are not conscious, and I can provisionally accept the idea that they are not on a path to consciousness.
I completely reject the notion that consciousness is the basis of human rights. The premise itself is absurd. We only have one unambiguous example of a class of conscious entities, and that it humans. If a human l […]
- qarl
Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"
Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".
Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".
Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".
Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
- moomin
Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
- io84
I’m sympathetic to OP but think this is a hopeless battle.
1 - The commercial demand for anthropomorphised models is already immense, pre AGI.
2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI.
I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI.
There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/
- hosel
>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
- addag
It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs.
Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
- binlog
Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.
You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.
Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
- redmaple892
Anyone know what the source of the header image is?
- NinjaTrance
> AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
That should be pretty obvious to anyone who ever created a chatbot using the top LLM APIs:
You can send the same question 1 million times to the same API, and it won't get tired from answering it. But if you simulate a conversation where the same question is repeated 10 times, it will auto-complete the text in a way that seems human. However: you can manipulate it by changing the conversation history; you can reset, roll back and branch the conversation at any point.
- simonw
Pet peeve:
> In a lengthy essay, Suleyman praised Anthropic boss Dario Amodei and his team for being "thoughtful, principled, and intellectually honest people" - but nevertheless questioned the company.
I wish people in mainstream technology publications would get better at LINKING to things. That "lengthy essay" needs to be a link.
UPDATE: I don't think this essay has been published yet? It's been "shared first with Axios", but I haven't been able to track down the actual essay itself.
Could it be this long tweet? https://twitter.com/mustafasuleyman/status/21002235945341504...
I don't think so, the essay in question is meant to have the phrase "hall of mirrors" in it, that tweet doesn't.
UPDATE 2: Found it: https://mustafa-suleyman.ai/a-warning-about-model-welfare - via https://thenextweb.com/news/suleyman-anthropic-claude-consci... who DID link to it.