Scott Aaronson: The Strange Loop That Wasn't — Why AI Didn't Need Self-Reference
LLMs and Self-Referentiality
In a provocative blog post, Scott Aaronson argues that the success of large language models has buried the once-dominant idea that self-reference and 'strange loops' are the key to intelligence. He points out that modern AI, from transformers to training data, contains no explicit self-referential machinery—yet models can discuss themselves and Gödel's theorem fluently. Aaronson suggests that self-reference emerges as a byproduct of universality, just as it did in mathematics and computing, and that the real drivers of intelligence are prediction and compression. He leaves consciousness as an open mystery but declares the self-reference-centric view of AI a wrong idea on par with phlogiston.
No one ever needed to build those abilities in, any more than self-reference needed to be built in to the alphabet or the rules of grammar.
- thatjoeoverthr
Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
But I suppose it doesn’t harm its reasoning!
- m-hodges
I've used concepts from GEB a lot in my thinking about how LLMs.
On self-referentiality and the MU Puzzle: https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...
On Gödelian limits of prompt-safe AI : https://matthodges.com/posts/2025-08-26-music-to-break-model...
On the blurry line between pattern matching and reasoning: https://matthodges.com/posts/2026-08-19-bongard-problems/
I'm glad that the author of this post ends with:
> What’s left? Consciousness
because when I've read (and re-read) GEB, I find the book to be much deeper than just a threshold for conversational intelligence.
- gwd
Two comments on this, trying to take a "which hypothesis fits the evidence" approach.
First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what it's read about them. In the same way, "non-aware spicy autocomplete" would produce an LLM that spoke about its own experience, based only on the input it has of people speaking about their own experience.
That said / secondly, from the little I understand of LLM architecture, I believe there are a large number of self-referential mechanisms built in. For one, nearly all transformers have a "residual layer", with various neural networks essentially reading and modifying it. This effectively forms a loop. Additionally, the "thinking" mechanism allows it to read what it's written and generate more things, which is again a loop.
So, maybe people didn't think, "Hey, we should build some loops, maybe that will make it conscious". But if "strange loop" is what defines consciousness, there are lots of loops in there onto which such a strange loop could conceivably form.
- macleginn
‘But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence?’ — has anyone ever formulated this actual idea or anything logically equivalent? This looks like a mighty straw man.
- killerstorm
Perhaps "true" self-referentiality was not needed. But it seems "dynamic feedback" is essential, and it seems to be adjacent to self-referentiality.
If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.
Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.
Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.