Cloudflare runs Kimi and GLM at scale with 8-bit caches and 4-bit weights
Smaller, faster, safer: running Kimi and GLM at scale
Cloudflare's Workers AI serves Moonshot's Kimi K-series and Z.ai's GLM models on GPUs close to users. To fit these large, long-context mixture-of-experts models into memory, they quantize the KV cache to FP8 and compress weights to INT4, doubling context capacity and boosting decode throughput by up to 55% with no accuracy loss. They also add cache integrity checks to protect shared memory, all while keeping overhead under 1%. These optimizations lower costs and increase capacity for customers.
Quantizing the cache adds a small amount of work per token, since the FP8 attention kernel has to convert values as it reads them.
- HDBaseT
I think Cloudflare not providing ZDR on their inference is the biggest public indicator that Cloudlare glows.
We let all traffic get MITM'd, now we're letting our AI conversation get tracked. Cloudflare reeks like a US Honeypot.
- scrlk
Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation.
However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time.
- syntaxing
> View pricing in the Cloudflare dashboard ↗
Why… I wanted to see if it’s worth it to use cloudflare’s endpoint but I can’t even see the pricing
- om8
Why int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.
- joshuamcginnis
> If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us.
What is the typical job title and/or skillset for this?