Apple M3 Neural Engine Bug Caps DRAM Bandwidth at 17 GB/s, Halving LLM Speed
Getting 50 GB/S Back from the Apple Neural Engine
An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming to 17–19 GB/s from the nominal 45–60 GB/s whenever total weight size is an integer multiple of 1 MiB. This affects 7 of ANEMLL's 15 models. Avoiding the problematic speculative prefetch ring in the kernel DMA engine raised Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s and Qwen3-8B from 1.36 to 2.97 tokens/s.
At D=2048, throughput was 16.93 GB/s. At D=2016, throughput was 44.5 GB/s, meaning 44.505062 − 16.930761 = 27.574301 GB/s (61.96% lower).
- bee_rider
Nice investigation.
It is always surprising to me when a nice round number like 1MiB results in the “bad performance” configuration (although it happens).
Are you sure erratum is the right word in this context? I usually see it used to describe the notice that a document has an error in it.
- thenewwazoo
"Apparently the memory controller's throughput has a dominant harmonic with wavelength 2048 in tensor-dimension space."
That got a laugh out of me.
- anemll
Not all systems affected
M1 and M5MAX are OK
- Neywiny
Just checking here- this systemverilog is a hypothetical telling of what you think is going on? Or do you have the actual source of the RTL?
- msephton
I guess any processes affected by this will solve this in software as time goes on?