Running a 28.9M Parameter LLM on an $8 ESP32 Microcontroller
Running a 28.9M parameter LLM on an $8 microcontroller
I managed to run a 28.9 million parameter language model on an $8 ESP32-S3 microcontroller by leveraging Per-Layer Embeddings from Google's Gemma models. Instead of loading the entire model into fast RAM, I stored most parameters in flash memory, fetching only what is needed per token. This approach allows the device to generate coherent short stories at roughly 9 tokens per second entirely on-chip, without any server connectivity.
The large model then costs almost nothing to run, because you never load most of it.
- titzer
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
- helsinkiandrew
Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
- rao-v
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
- kamranjon
Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.
- NooneAtAll3
While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights