Running the 2.78 Trillion Parameter Kimi K3 Model on a Consumer Laptop
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
I built WASTE, a dependency-free C inference engine that streams activated weights directly from NVMe storage. This allows the full 2.78 trillion parameter Kimi K3 model to run on a standard 64 GB MacBook Pro without fitting entirely in RAM. By keeping only the necessary expert weights in memory and streaming the rest, we achieve local inference at roughly half a token per second, proving that frontier-scale models can operate on consumer hardware.
The interesting part is not the speed, it is that the whole thing is in the reachable range on a single consumer machine — and that from here the question is engineering rather than feasibility.
- ikurei
They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...
I hope that reasoning is an after-the-fact justification by the LLM that wrote this.
It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
- theanonymousone
For me at least, it makes way more sense on a DSv4 Flash size and ~10 tps and I would definitely try (or try to try) such a thing.
The concept and the proof of it is great, of course.
- bgirard
Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.
- nialv7
are they allowed to use the "SQLite" name?
- hddambo
Pretty soon we'll have 3T param models down to 1 bit. They'll be able to tell you whether they're off or on.
- mappu
Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
- pja
That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
- bilsbie
Claude might as well be .5 tok/s. I end up waiting several minutes and what it tells me could usually be summarized in under 100 words.
So I could potentially live with this if it was concise.