MicroGPT in Pure C Hits 10M Tokens/sec on Apple M5
Microgpt in pure C hits 10M tps on Apple m5
A new GitHub project, microgpt-c, implements a character-level GPT entirely in dependency-free C, with training and inference in a single file. It achieves over 10 million tokens per second on an Apple M5 Pro using NEON optimizations, and 6.9M on an AMD Ryzen 5 5600H with AVX2. The model, with only 4,192 parameters, generalizes well, beating an interpolated trigram on unseen data.
c fp32+NEON 10168430 tok/sec
- hasteg
This is pretty cool. Implemented something similar myself (a really small language model with ~10M params) just to teach myself the ML behind the LLMs. Did not implement it in C obviously, just use PyTorch, but it's interesting to go through the c file and see how he has implemented stuff I took for granted in Python in C.
Anyway, I just tested this out myself on my AMD Ryzen 9 9800x3d. I got 7647173 tok/sec using karpathy's Shakespeare dataset https://raw.githubusercontent.com/karpathy/char-rnn/master/d.... Going to play around with it and see if I can get a CUDA kernal built to see what it could do on a 5090. Claude estimates with napkin math that we could get around 2B tok/s
- Retr0id
> The most atomic way to train and inference a GPT in pure, dependency-free C.
What sense of the word "atomic" is meant here?
- MycroftJones
Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version. https://github.com/moebiusV/cugpt
- ilaksh
This is not an LLM obviously
, it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
- gok
It could probably go quite a bit faster using Arm SME. The entire network could fit into the ZA register.