A Visual Guide to Quantization: How to Shrink LLMs Without Crushing Accuracy

A Visual Guide to Quantization (2024)

Large Language Models (LLMs) are often too large for consumer hardware, but quantization can shrink them dramatically. This visual guide explains how floating-point numbers are represented in bits, then walks through symmetric and asymmetric quantization, including absmax and zero-point methods. It covers the trade-off between precision and memory, the problem of outliers, and calibration techniques for weights and activations. With over 50 custom visuals, it demystifies how to compress models like Llama 3 from 280GB down to a fraction of the size while keeping performance intact.