Speculative Decoding Is Fast Because It Wastes Compute on Purpose
Why Is Speculative Decoding Fast?
Speculative decoding doesn't reduce the work per token — it increases total FLOPs. The speedup comes from shifting the GPU from memory-bound single-token decoding to compute-bound parallel verification, using otherwise idle compute at small batch sizes. At high throughput the trade flips: rejected draft tokens become pure waste, which is why vLLM lets you disable speculation above a batch-size threshold.
When LLM decoding is memory bound, you're wasting compute. The GPU could be doing more operations, but it's not.