50 tok/s on a 24GB GPU: How I squeezed Qwen3.8 27B to 256K context
Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
A developer details how they achieved 50 tokens per second with Qwen3.8 27B at 256K context on a 24GB RTX PRO 4000 SFF GPU. The key was a custom quantized model that used NVFP4 for bulk matrices and higher precision for sensitive layers, plus an embedded MTP drafter and a patched llama.cpp build. The winning setup was not about any single best component but the fit between quant, drafter, kernels, and memory layout.
The winning setup came from the fit between the quant, drafter, CUDA kernels, memory layout, and workload. No component won on its own.
- simonw
"Combining them into one heroic speedup would make a better headline and a worse benchmark."
"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
- supermatt
Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.
- gitowiec
Why it's flagged?
- Tepix
Always put the quantisation in the title!
- nodja
The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).