I Trained a Tiny Language Model to Run on a 1975 8-Bit Processor
Autoregressive Language Model on the 6502 Processor
Matt Beton trained a BitNet-based autoregressive language model small enough to run on his dad's BBC Micro, a 1980s computer powered by the 8-bit MOS 6502 processor with only 32KB RAM. The model uses ternary weights and a Mamba-style recurrent architecture to fit within 13KB of weights and 9KB of inference code, generating text like 'once upon a time tom and lily saw things lily were sad...' directly on the vintage hardware. The project tackles challenges like the lack of a multiply instruction and memory constraints, and includes a browser-based emulator for readers to try it themselves.
The CPU only operates on an 8-bit integer datatype, and doesn't include multiplication in its instruction set.
- bmc7505
Cool to think this demo would have been possible over fifty years ago. I wonder what someone from 1975 would have said if you had shown this to them back then.
- toplinesoftsys
This is amazing project! I hope it will result in real miniaturization of AI - for example, edge LLM inside of glasses. That will be awesome.
- tyromaniac
This is super cool!
As someone who's worked a little with NES programming and tried out cc65, I'm surprised he didn't just hand write some assembly, he likely couldve saved a lot of space if I had to guess.