Apple's Neural Engine Was Built for a World That No Longer Exists
Retrospectively Reverse-Engineering Apple's Neural Engine
Three years after abandoning his reverse-engineered ANE driver, the author returns to dissect the M1 Neural Engine's internals. He maps the 16 compute cores, 2048 MAC lanes, fixed-point accumulators, and 33-entry tanh lookup table, revealing an architecture optimized for CNN dataflow that transformers broke. With the M5 folding ANE cores into the GPU, this teardown doubles as a eulogy for the standalone NPU.
The assumption that transformers broke, especially with autoregressive decode, was predictable reuse patterns, which the ANE exploited to architect a dataflow efficient enough to run on phones.
- zozbot234
How does this relate to the more recent work on the M4 ANE found at https://maderix.github.io/articles/ ? Does the M4 and later ANE expose any additional capabilities, or is it just a higher-performance iteration of the same thing?
As an aside, the introduction to this article seems to conflate the ANE with the Neural Accelerators (NAX) found in the M5+ (and A-series equivalents) GPUs. These are very different things, and Apple is still working on the ANE - the M6 and A20 will apparently feature doubled ANE blocks.
- GeekyBear
It's worth remembering that Apple is releasing a new framework (Core AI) this fall that goes beyond the Pytorch and Tensorflow workloads that the decade old Core ML framework allowed.
> Core AI allows your app to use the latest model architectures and inference techniques across the CPU, GPU, and Neural Engine.
- throw0101a
A lot of folks are/were saying that Apple has missed the boat when it comes to AI, in some (important) aspects that is correct, but I think it's worth remembering that Apple added the Neural Engine to A-series chips in 2017, before the AI hoopla really kicked off:
* https://en.wikipedia.org/wiki/Neural_Engine
* https://apple.fandom.com/wiki/Neural_Engine
"AI" has grown much more since then, and there have been important developments that Apple has not deployed (well), but I think they were looking ahead a little more than most at the time (even if events 'got away' from them subsequently).
- anentropic
Amazing analysis
Same author even found a bug in it https://eiln.github.io/posts/ane-dma.html
- CraigJPerry
This isn't ai slop. It's fascinating and well written.
But I learned something really basic - i didn't know that the ANE (and the data pipeline around it) was designed for CNN rather than transformers. It's always been an open loop in my head, wondering why the ANE was less impactful than i understood it should be.