Swiftlet - Run 80B Qwen in 4.3 GB RAM on Mac, 35B on iPhone
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Swiftlet is a Swift + Metal runtime that brings Qwen3-Next and Qwen3.5/3.6 MoE hybrid models to ordinary Apple devices, including iPhones. By keeping only the small dense core resident and streaming routed experts from storage on demand, it runs an 80B model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone. With runtime-compiled Metal shaders, it achieves 4.5-11 tokens per second on M5 Macs. The project includes a CLI, an OpenAI-compatible server, and an open-source iOS app (Priv AI) for on-device chat. Built with Claude Code, it's a breakthrough in local AI accessibility.
As far as we know, that is the first time a model of this class has run natively on a phone.
- dghlsakjg
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
- AHASIC
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
- gitpusher42
Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README.
I am glad it inspired more people to explore area of on-device AI further!
- myshapeprotocol
Achieving high-efficiency model compression to run large models locally on consumer hardware like Macs and iPhones mirrors the architectural goals of decentralized identity.
- gizmodo59
The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
- adrianco
This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
- CyLith
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
- sallymander
"As far as we know, that is the first time a model of this class has run natively on a phone.".
I feel like I've seen a similar statement on a lot of these streaming weight projects.
400b model on an iPhone: