Run a 125B AI model on your gaming PC at 100 tokens per second
Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

Strata, an open-source inference engine, runs Qwen3.8-Flash-Next on consumer GPUs with 12GB VRAM. It splits the model's 24,576 experts across GPU, RAM, and SSD, achieving up to 94 tokens/s on an RTX 5070 and an estimated 100-140 tokens/s on an RTX 3090. One-click install for Windows and Linux, with OpenAI/Anthropic-compatible API on localhost and optional image input.
Strata makes the model fit by sharing the work across your whole PC.
- a11r
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
- Jackson__
I've just tested Strata on a simple 50 image vision benchmark. The task is to output the exact coordinates of a requested object. The result via Strata had a median error distance of 154.8 pixels, avg of 168.8. Running the exact same GGUF and vision adapter weights on llama.cpp gives me a median error of 46.5, avg 81.4.
To put that into perspective, here are some more numbers from other models via llama.cpp:
Median/Average
Qwen 3.5 9B BF16: 46.5 / 193.3
Qwen 3.6 35B Q4 K XL: 38.4 / 76.4
Qwen 3.5 122B Q3 K M: 32.9 / 68.6
The difference in vision performance is as large as the jump from a 9B model to a 35B model. All tests were performed at temp=0.
I have done no further testing, as these results line up perfectly with my expectations.
- XCSme
For what's worth, on high, 27b is better, but both models on high are too slow to be used (way too many output tokens).
On low, 3.8 Flash Next is better AND considerably faster.
My results[0][1], on a RTX 3090 + 128GB DDR4 RAM. I still have to see if I can optimize any more settings for either, but I think I will switch from 3.8 27b to 3.8 Flash Next for my local LLM uses.
[0]: https://aibenchy.com/?q=qwen+3.8+27b%2C+qwen+3.8+next
[1]: https://aibenchy.com/compare/qwen-qwen3-8-27b-low/qwen-qwen3...
- snehesht
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
- cjdell
This is game changing. My R9700 32GB is now smarter and about 2x faster than using Qwen-3.8-27B. About 60 t/s when combined with my 96GB of DDR4. My motherboard limits me to PCIe Gen3 so that is likely a bottleneck.
For the Nix inclined:
https://github.com/cjdell/nixos-config/blob/main/hosts/zen3-...
Even got it running on the iGPU of a GMKTec M6 Ryzen 6600H at reasonable speed (10 t/s). Fast enough to leave it with a prompt before I go to bed and wake up to a solution.
- AntiRush
I've been working on support for this model in ds4 on the RTX 6000 pro - it's been really great for my use cases. The ds4 q4 quant performs a lot better than other similar sizes that I've seen.
Using the Q4 quant on an RTX 6000 Pro Workstation Edition at 450 watts:
Code: prefill 1,251 tok/s decode 255.26 tok/s
Prose: prefill 1,251 tok/s decode 198.78 tok/s
Most important for me, I can run 4 concurrent streams at 400+ tok/s.
- kamranjon
Dwarfstar already supports this, curious how it compares, but I use the q4 quant daily and it works really well.
https://github.com/antirez/ds4/blob/main/docs/MODELS.md#qwen...
- jacquesm
LLM threads the world over are spammed with Strata links, it remains to be seen how much of the breathless hype remains standing once the honeymoon period is over. I've tried it but so far I have not seen anything that overly impressed me in terms of accuracy, though the speed is definitely there. I'm sure there are applications for LLMs where the quality of the answers is less important but I don't have any of those. YMMV.
- lxe
Why isn't this type of expert caching in the native llama.cpp yet? Why do we need a separate codebase?
- SuperV1234
We're getting closer and closer to the day we can have an Opus-like model running locally. The dream!