TurboFieldfare - Run Gemma 4 26B on M-series Macs with 2 GB RAM
Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
TurboFieldfare is a custom Swift and Metal runtime that enables the Gemma 4 26B-A4B model to run on any Apple Silicon Mac, even those with only 8 GB of RAM. By keeping the shared core in memory and streaming expert weights directly from the SSD, it achieves inference with a mere 2 GB memory footprint. This open-source project offers a native Mac app, a CLI, and an OpenAI-compatible server, allowing developers to leverage powerful large language models without expensive hardware upgrades. Built specifically for macOS 26 and Metal 4, it bypasses generic wrappers to deliver optimized performance for local AI tasks.
Memory got expensive. So I gave a 26-billion-parameter model a ~2 GB budget.
- giancarlostoro
Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.
Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. I wouldn't be surprised if frontier AI could be drastically cheaper if they just finetune and optimize their models to not consume all available RAM to only access less than 10% of the models knowledge.
- pwython
Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea.
Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s.
Page cache seems to be why it beats the M5 Pro. With 64 GB the whole 12 GB packed_experts set stays resident, and iostat shows only ~1.6 GB per run actually reaching disk, against the ~79 GB that 98 fully cold tokens would need.
I then tested with DaVinci Resolve open and under load (playback): 42.6 tok/s. Also held 38 GB of incompressible memory to squeeze the page cache: 41.8. At 48 GB it ranged 32 to 41.5. Degrades gradually rather than a cliff. It's a beautiful thing.
- xenonite
With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with
opts.languageVersion = .version4_0
or surround them with
if #available(macOS 26.0, *) {
opts.languageVersion = .version4_0
}
You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)
- tredre3
I'm curious how your project compares to plain mmap!
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
- mmastrac
I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other.
Feel free to reach out.
(currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)
- woadwarrior01
> The measured result is a reference point, not a performance ceiling.
Claude was here.
- ycui1986
There are a lot of SSD streaming engines these days. But few to actually try some hard features.
There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.
- nvch
12 tok/s and almost instant response on M1 Max Mac Studio (with faster SSD than laptops) are impressive – gives hope that large models may run locally from SSDs instead of memory.
- addaon
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
- owaislone
Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.