Qwen3.8-27B runs on the iPhone 18 Neural Engine at up to 70 tokens per second
Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)
Anemll demonstrated a 27B-parameter Qwen3.8 model running locally on the iPhone 18's Apple Neural Engine, hitting 15–70 tokens per second. The base is PrismML's Bonsai-2 at 1.585 bpw, requantized to 1.5 bpw with 6-bit vector quantization per four weights, plus a dflash-2 draft model at ~2.8 bpw for speculative decoding. A follow-up confirmed 24K context with int8 K/V cache while the delta net stays FP16.
Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context. Apple Neural Engine on A20 is fast 🚀
- Retr0id
OP, asking for upvotes is against the HN guidelines: https://x.com/anemll/status/2111531629529456771
- analog_daddy
Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.
App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.
- createful
Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.
Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.