MAPLE - Sub-2-BPW MoE pruning for running Gemma 4 26B on 6GB GPUs
Show HN: Maple – Sub-2-BPW Moe Pruning to Run Gemma 4 26B at 45 TPS on a 6GB GPU
MAPLE is a domain-agnostic pruning pipeline for Mixture-of-Experts LLMs that slashes Gemma 4 26B-A4B from 9.92 GB down to a 4.75 GB GGUF container (1.48 effective bits per parameter) by profiling real forward-pass expert activations and excising 80 dormant experts per layer. The result: 100% in-VRAM execution on a 6 GB laptop GPU like the RTX 4050, hitting 45.4 tokens/second — a 6.57x speedup over the offloaded base model — while retaining 93.8% semantic fidelity on code synthesis, data structures, physics reasoning, and formal logic. The pipeline is reusable for any domain via calibration, selection, and container assembly scripts, with a pre-built checkpoint available on Hugging Face.
Compressing Gemma 4 26B-A4B from 9.92 GB to 4.75 GB, enabling 100% In-VRAM GPU execution on 6 GB laptop GPUs with 45.4 tokens per second.
- aetherspawn
Does it improve tps for Mac?
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
- viraptor
That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
- hxii
Can the same process be theoretically repeated with LFM2.5-24B-A2B ?