Ternary LLMs waste bits on zeros, and a new layout reclaims them
Breaking the 1.58-bit Barrier for Ternary LLMs
Ternary LLMs store weights as {-1, 0, +1}, but standard five-trit packing assumes all three symbols are equally likely and costs up to 1.625 bits per weight. Measuring 29 ternary models reveals zeros make up as much as 51.5% of weights. BITCOS, a distribution-adaptive layout using a presence bitmap and compacted sign vector, stores weights more compactly in 26 of 29 models, reaching 1.485 bits per weight and up to 1.28× faster matrix-vector multiplication, with decode throughput gains up to 1.18× on CPUs and 1.27× on GPUs.
We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to 51.5% of all weights.