Neural Networks Are Already Clean, Argues Mechanistic Interpretability Researcher
After months studying neurons in image and language models, a researcher concludes that polysemanticity and superposition were misread. What looked like arbitrary packing is mostly recursive generalization, a fractal form of ordinary generalization. In Qwen3-32B, roughly 90% of neuron poles have one main job, and only 2-4% remain unexplained. The canonical InceptionV1 example now shows 99-100% interpretable neurons in its last layer. This means we can study networks directly, without first untangling them.
It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition.
- sebmellen
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
- bluejay2387
The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.
- QuadmasterXLII
this article seems to take it on faith that neurons are aeasily findable, even if their meaning is tricky to untangle: it would be sharply improved by description of how the neurons are being isolated before the meat of the article, the naming of the neurons, is performed. In particular axis aligned semantic directions are incompatible with the classic superposition description, where the meaningful directions are nearly orthogonal, allowing many more to be packed onto the sphere than it has dimensions. (i’m also aware of work showing that Adam or relu pushes semantics onto the axes, but it’s not clear that that’s what he thinks is happening)
- lukewarm707
It reminds me of this quote from Judith Butler:
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
- aesthesia
The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.
- ks2048
To cut through some of all this text - here's a link he calls "the first non-trivial neural network I feel I mostly understand:
https://already-clean.goodfire.com/microscope/medium.html?la...
- juanjmanfredi
Yudkowsky seems unconvinced.
- x312
A TLDR, since most comments here seem to misunderstand the result.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.