Two NVIDIA DGX Stations Serve a 753-Billion-Parameter Model at 607.8 Tokens per Second
Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

GLM-5.3 NVFP4 packs 753 billion parameters into a 464 GB checkpoint, too large for the 252 GB of HBM3e on a single GB300 GPU. NVIDIA's walkthrough links two DGX Stations over ConnectX-8 RDMA and runs vLLM with tensor parallelism, splitting weights and computation across both GPUs behind one OpenAI-compatible endpoint. Multi-token prediction, FP8 KV cache, and prefix caching push throughput to 607.8 tokens per second, with setup, verification, and load-testing steps included.
At this point one model spans two machines. Neither GPU holds a complete copy, and neither can answer a request alone: every token needs both ranks to exchange partial results over the RoCE rail.
- hkalbasi
Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity. Which makes it economically unreasonable unless you want privacy or something like that.
- jbellis
This doesn't seem all that impressive for $200k of hardware? You could serve 4bit off 8xB6000 for ~half the price. No idea what throughput you'd get tho so maybe that's the flex.
Edit: This guy already did it, and you get 60% more throughput on the 8xB6000 https://github.com/kacper-daftcode/GLM5.3-8xPRO6K
- k0morebi
Is my intuition correct in that the headline tok/s are for a single user?
Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?
- arjie
Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
- AnotherGoodName
For anyone actually considering this $200k+ setup for home or even small business use the power usage and noise is extreme. 1.6kw each with the fans to match. 600tokens/s at concurrency 16 is not what you need for home use or even sub 100 employee business use either. Really reconsider please (do you seriously need people to tell you this?). I get tech wages are high and you can probably afford this but seriously.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory […]