Cloudflare's Cache Transcoding could save petabytes of storage with Zstandard
We could save petabytes of cache storage with Zstandard and Pingora
Cloudflare prototyped Cache Transcoding, which uses Zstandard compression inside Pingora to shrink eligible cached assets to about a third of their original size. The trade-off is a small CPU increase for significant storage and bandwidth savings, especially for text-based content. Tests with over a million requests confirmed the architecture's viability, and the team plans to explore higher compression levels and broader content types.
A small increase in CPU gives Cloudflare petabytes of effective cache capacity and reduces the data transferred between our data centers.
- r3trohack3r
Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files.
For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.
There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
- thinkindie
Why not serving files compressed if the client supports it even though the origin served an uncompressed file?
- CodesInChaos
I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.