Compression Is Prediction: How LLMs and Compressors Solve the Same Problem

Compression and language modeling are fundamentally the same task: predicting the next symbol. This post breaks down the anatomy of a compressor—transforms, models, and entropy coders—and explains arithmetic coding, a technique that encodes an entire dataset as a single number. The key insight: better probability predictions lead to better compression, and the same principle drives LLMs. With a concrete example, the author shows how skewed probabilities shrink average bits per symbol, revealing the deep connection between compression and AI.