Git at Any Scale: Why Hosting Repositories Is a Nightmare
Git's distributed design makes hosting repositories at scale a nightmare. Packfiles, the binary format for storage and transfer, create random read patterns that break networked filesystems and distributed stores. GitHub's early attempts—NFS, GFS, DRBD—failed, leading to Spokes, a consensus-based replication system that keeps full copies on NVMe drives and ensures strong consistency. This article explores why packfile-based design is the core challenge and how Spokes became the industry standard.
Packfiles are the fundamental building block of Git storage and Git networking.
- gritzko
The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.
- wrs
There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.
- brasic
It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.