Training a Transformer from Scratch on an $8 ESP32-S3
An SLM trained on $8 ESP32-S3
This project demonstrates training a character-level transformer from scratch on an ESP32-S3 microcontroller, with backpropagation written by hand in C. The model, Qapla', learns Klingon from a small corpus, achieving real learning on-device without any external computation. It highlights the feasibility of on-device training for edge AI, constrained by memory and model size, and explores potential applications like predictive maintenance and soil monitoring.
The chip starts with random weights and no prior knowledge of Klingon, and learns from scratch.
- chicken-stew
Would probably be better to demonstrate by exsmple how this approach is used to train on sensor data and then use it (as is hinted by the author) instead of acknowledging that the klingon poc is useless.
- runtime_lens
Sometimes the proof of concept isn't the product. It's the constraints it exposes that end up influencing more practical systems.
- dannyw
Very cool project! Sounds like it was a fun challenge :)
I wonder when we'll start seeing clusters of ESP32-S3s... not sure how interconnects would go though, but I guess the interconnect wouldn't be the bottleneck anyway.