INT8 ConvRot: The New Standard That Makes FP8 Obsolete
Explanation of INT8 ConvRot (FP8 is no longer needed)
INT8 ConvRot, natively supported in ComfyUI v0.27.0, is a new quantization method that combines INT8 encoding with a rotation-based technique to handle outliers. It outperforms FP8 and FP8 Scaled on RTX 40/50 series and offers significant speedups on older RTX 20/30 series, which lack FP8 hardware support. This article explains the method, its performance, and how to use it in ComfyUI and Forge Neo.
INT8 ConvRot will become the standard for all 8-bit quantized models.
- gopalv
The other CloudFlare post on the front page has an interesting passage in it
> It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins.
So each of these improvements are useful even if they have a narrow area of applicability, since the systems can be hybridized for performance.
[1] - https://blog.cloudflare.com/smaller-faster-safer-models/
- p1esk
Is there any degradation with INT8 weights quantization? Why would anyone want to apply ConvRot to do 8 bit weights? Note the paper [1] focuses on 4 bit weights and 4 bit activations (W4A4) quant scheme - a much more challenging goal. My understanding is that W8A8 with weights quantized per-output-channel is a solved problem - does not need any fancy methods like GPTQ, SpinQuant etc.
- randomblock1
What about AMD? I'm guessing it's not supported, which is a shame, because they're better value for VRAM.