A 284B DeepSeek model now runs on a 64GB CPU with zero GPUs
Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU
llama-modchip squeezes DeepSeek-V4-Flash, a 284B-parameter MoE, from 76.87 GiB down to 56.56 GiB, so the whole model plus a 256K context fits in 64GB of RAM on a plain CPU. It hits 44.4 tok/s prefill and roughly 9–14.7 tok/s decode, using SwiGLU-H2 Hessian compression, Fractal-64 MLA transcoding, and AVX-512 tiling. Smaller 30B MoEs reach up to 111 tok/s.
Real-time Frontier & Agentic MoE inference on the CPU workstation you already own — 100% in RAM, zero NVMe swap.
- kennywinker
The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.
I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?
- aappleby
Stupid question, but does this AI-generated repo actually _do_ anything?
Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in "modchip setup", so I haven't tested that one yet.
Edit: this is on a Ryzen 7945hx 16-core Zen 4 + 96 gb DDR5.
- armcat
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
- sparse-Matrix
It works pretty good on my Ryzen 7 5700U/64GB DDR4
Steady 12 to 14 TPS.
Not blazing, by any means, but fast enough to get meaningful personal work done, with a power consumption of 15w.
- antirez
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175
Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
- boredatoms
The quantization is probably the most important thing to know and its not clear at all from the dense word salad
- akie
At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
- SillyUsername
Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.