Gzip can write Shakespeare, sort of
Can gzip be a language model?
The compression–prediction equivalence says every compressor is a predictor. So I built gzipt, a pure-Python language model that uses zlib's DEFLATE to score continuations by compressed length, then generates with beam search. Primed on Tiny Shakespeare, it produces semi-coherent text—no neural network, no training, just the compressor in your OS.
A continuation that gzip “expected”, because it echoes text already in its window, compresses to almost nothing.
- aghilmort
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
- jll29
Yes: you can classify a test file by topic with gzip as follows:
gzip -9 sports.txt testfile.txt
gzip -9 politics.txt testfile.txt
gzip -9 business.txt testfile.txt
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
- Culonavirus
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
- mg
give it a normal text prompt, and it
continues that prompt by searching
for the byte sequences that compress
best.
One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
- GodelNumbering
3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
- _def
Fun topic but generated article text and then not even actually using gzip? Rubs me in a weird way.
- montebicyclelo
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
- colinmarc
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.