uv's new file-level deduplication trims 545 MB from the wheel cache
uv: Deduplicate all files in the wheel cache
A pull request by Charlie Marsh introduces file-level deduplication to uv's wheel cache. Instead of only deduplicating entire wheels, every file is now stored under its BLAKE3 hash in a `files-v0` bucket, with hardlinks into the original archive locations. This saves about 545 MB (10% of the cache) on the author's machine, with a modest <4% slowdown on cold installs and no impact on warm ones. Benchmarks confirm all regressions stay below 5%.
So we're saving 545.2 MiB on my local machine, or about 10% of the cache.
- notatallshaw
As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
- mark_l_watson
Nice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use.
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.
- stephenlf
uv is the backbone of any modern Python library. I’m excited to see improvements.
- CivBase
A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.
- TacticalCoder
> deduplication at the file level: every file is now stored under its BLAKE3 hash
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
then:
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.