From Github · · 4m
Reverse-engineering Apple's Neural Engine, three years late
A hacker goes back to the M1's Neural Engine to map what Apple committed to silicon in 2017, and why transformers broke it.
In brief
A hacker goes back to the M1's Neural Engine to map what Apple committed to silicon in 2017, and why transformers broke it. The ANE's limit was never its math. Its memory paths assumed weights get loaded once and reused forever, which is exactly what token decode refuses to do. Originally reported by Github.
Quitting, then coming back
She dropped the reverse-engineered ANE driver three years ago after deciding the block just was not that useful. Even macOS mostly uses it to make Finder preview thumbnails.
“the ANE block is just not that useful, and I could be doing more useful things”
The M5 buried the NPU
Apple's M5 led with LLM performance and folded the ANE cores inside the GPU cores. The standalone neural processor looks finished as a separate block.
“it officially feels like the beginning of the end for the standalone NPU”
A different goal this time
Not making it useful, but mapping it: compute, datapath, scheduler, memory, execution model. Those choices record what Apple believed about ML workloads in 2017.
The MACs say nothing
Sixteen cores, each with 128 FP16 or 256 INT8 multiply-accumulate lanes, 2048 lanes total. A dot product is a dot product, so the array alone reveals no workload bias.
“A dot product is a dot product, and a MAC does just that.”
Dataflow is the real design
What specialized the ANE for CNNs was when and where MAC inputs and outputs enter, stay, and move. Transformers broke the assumption of predictable reuse, not the arithmetic.
“What specialized ANE to the 2017 CNN models is not the MAC, but dataflow surrounding the MACs”
Time is the only reduction axis
Each lane keeps a running sum in a local 32-bit accumulator, so partial sums never travel to far memory. A lane has no idea which tensor dimension it is reducing.
Where the accumulator gives up
Feeding a vector of ones shows the output pinning to infinity at 32768, a value FP16 can represent fine. So the clamp is inside the accumulator, a Q16.16 fixed-point register saturating at 2^15.
tanh is a lookup table
Compiling a single TANH layer leaves 33 FP16 words in the register file. They match quantized samples of tanh(i/8), covering 0 to 4 with 1/8 spacing.
Proving the interpolation
Build a table with one spike at index 8 and sweep the input. The output draws a clean triangle rising to 1 then falling, so mode 2 is a piecewise-linear LUT, not 33 hard bins.
“The measured output forms a triangle”
Scale and bias get folded
A ReLU with a constant scale and offset shows the compiler rewriting the convolution weights instead. Authored W=4, b=-2 becomes compiled W=2, b=0.
The driver is boring
It never sends a CONV or MATMUL opcode. It writes a task address and size, then rings a doorbell, and hardware owns the job until an interrupt says it finished.
“The ane driver source code is disappointingly boring.”
Eight queues, ping-pong slots
Eight identical register blocks hold priority, state, and two command-stream slots each, so one slot runs while software loads the other. Much like a GPU hardware channel.
No ISA at all
A task descriptor is not an instruction stream. It is a burst of DMA writes into hardware config registers, a serialized dump of the datapath's own register file.
“ANE has no ISA.”
Configure, do not program
Each ANE program is one pass through a fixed datapath. You can set the knobs it exposes, but you cannot add operations or reorder how they are sequenced.
“ANE is a fixed-function dataflow engine, not a GPU executing arbitrary instructions.”
Static shapes are not the problem
A growing KV cache can just be looped over, and dispatch overhead is tiny. Dynamic execution is not what holds the ANE back.
“What actually shaped ANE for CNNs over transformers is memory movement.”
162 operations per byte
11 TOP/s over 68 GB/s sets the roofline ridge. Every DRAM byte must feed at least 162 operations, or faster math buys you nothing in tokens per second.
“Below the 162:1 ratio, speeding up compute won't increase decoded token/s.”
Weights got their own road
Three DMA engines: kernel loads, tile in, tile out. Kernel memory is 64 KiB private SRAM per core, load-only, because weights were assumed to be fetched rarely and reused constantly.
The missing L2 to KMem path
Weights can only come from DRAM, never from the shared 2 MiB L2. She argues the mux would have been cheap, and Apple simply never expected an L2-resident tensor to become a kernel.
“It makes me think that Apple simply never expected L2-resident tensors to become kernels”
Double-fucked bandwidth
Measured reads: kernel DMA 38 GB/s, tile DMA 59 GB/s, GPU 78 GB/s. Worse, running kernel and tile together takes the sum of their times, so the requests are serial.
“ANE's kernel and tile DMA requests are sent serially”
Only bandwidth can save it
With decode pinned to the memory roofline, going from 10 to 25 tokens per second needs roughly 2.5x more streaming bandwidth. The post ends hunting an extra 50 GB/s.
“But … what if we could add 50 GB/s of additional kernelDMA throughput?”
The bottom line
The ANE's limit was never its math. Its memory paths assumed weights get loaded once and reused forever, which is exactly what token decode refuses to do.
