Running a 28.9M Parameter LLM on an $8 ESP32 Microcontroller
Running a 28.9M parameter LLM on an $8 microcontroller
I managed to run a 28.9 million parameter language model on an $8 ESP32-S3 microcontroller by leveraging Per-Layer Embeddings from Google's Gemma models. Instead of loading the entire model into fast RAM, I stored most parameters in flash memory, fetching only what is needed per token. This approach allows the device to generate coherent short stories at roughly 9 tokens per second entirely on-chip, without any server connectivity.
The large model then costs almost nothing to run, because you never load most of it.
- rao-v
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
- titzer
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
- chrishynes
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?