Huawei's MACaQ quantizes a 122B-parameter model in 64 minutes on 32.5 GB of memory
Huawei releases MACaQ, its internal quantizer for local inference
Huawei's research team open-sourced MACaQ, a model-agnostic calibrated quantizer that produces high-quality GGUF and W4A4 checkpoints for text, image, video, and audio models. It runs across llama.cpp, vLLM, Diffusers, and PyTorch, and on Ascend hardware. The project claims lower perplexity than MR-GPTQ and DuQuant++, or matching Llama-3-8B quality with 11× less quantization time, all under Apache 2.0.
MACaQ can quantize a 122B-parameter model in 64 minutes using just 32.5 GB of device memory.
- lukasc-ch
People are reproducing it and creating GGUFs https://www.reddit.com/r/LocalLLaMA/s/36atJ2nNjh
- Rabbit504030201
https://huawei-csl.com/blog/macaq/
Why does some HN users seem to prefer GitHub page whenever available at expense of official site? I mean, Huawei is still sanctioned, anything they host third party outside China can disappear at any hour.
- mocchibird
Did they release any model? Seems good for Strata