Optimizing Lua String Literals to Save 400 Bytes in ComputerCraft
My girlfriend and I faced a disk space crisis for our ComputerCraft programs, where our Lua code simply wouldn't fit on in-game diskettes. To solve this, we developed a clever compression strategy using raw strings and a custom bit-packing scheme to handle carriage returns efficiently. This approach allowed us to create a self-decompressing archive, saving crucial bytes while keeping the installation process simple for users.
Cobalt folds newline sequences into single LF in raw strings, which is like running dos2unix on your PNG and expecting it not to be corrupted afterwards.
- wahern
I built a benchmark harness for exploring compression of Lua script source files embedded in a Lua app binary as individual static constant C arrays. I settled on gzip, specifically deflate/inflate. Deflate does as well or better for small source files compared to xz, bzip2, and zstd.[0] More importantly, the inflate algorithm is tiny[1]; embedding the zstd decompressor blew up the binary.
I also explored building precomputed dictionaries, which means you can easily embed the files individually (C source file inclusion and runtime loading through the Lua C API remains relatively straight-forward compared to gymnastics of parsing and transforming preprocessed files) while getting similar compression ratios as when compressing an enormous file (e.g. a concatenation of all the source files). This is trivial with the zstd reference utility. It's also simple for deflate, though you have to roll your own dictionary builder by hacking the zlib implementation, or just writing it from scratch[2]. But I never bothered implementing it beyond the benchmark harness. I probably would have stuck with deflate rather than switching to zstd just because the compiled inflate implementation is so small.
[0] This is because the greatest advantage of the alternatives over deflate is the larger dictionary size they build up; deflate has a very small, upper bound. But for short inputs this advantage is diminished.
[1] https://github.com/madler/zlib/blob/develop/contrib/puff/puf...
[2] https://blog.c […]
- Rygian
> 125 KB of disk space on a diskette.
> I use SI prefixes for bytes. You should too!
The prefix for 1000 is lowercase.
- XJ6w9dTdM
I do something similar, embed large compressed data blocks as Lua string literals in some project, and I've just given up and used
lua_string_literal = string.format("%q", compressor(data))
I don't know if that's available in Cobalt, but for standard Lua I like the compromise, it's not /that/ wasteful.