Git at Any Scale: Why Hosting Repositories Is a Nightmare
Git's distributed design makes hosting repositories at scale a nightmare. Packfiles, the binary format for storage and transfer, create random read patterns that break networked filesystems and distributed stores. GitHub's early attempts—NFS, GFS, DRBD—failed, leading to Spokes, a consensus-based replication system that keeps full copies on NVMe drives and ensures strong consistency. This article explores why packfile-based design is the core challenge and how Spokes became the industry standard.
Packfiles are the fundamental building block of Git storage and Git networking.
- brasic
It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
- biwills
> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
- iamandoni
I have to say, when I first heard about Origin, I was skeptical. But now I do believe that the team behind it is capable of pulling it off and shipping something truly incredible.
This is one of the best technical articles I’ve read. And I came away a lot more knowledgable about a technology I have used daily for over a decade and consider myself adept at. Kudos to the author.
- nikolay
There's nothing Cursor can do that GitHub/Microsoft can't in 2026... And vice versa... after several years of Cursor vibecoding a GitHub clone while catching up with GitHub's new features. Git's performance it's not the issue; it's GitHub Actions, PRs, etc. The moment Cursors steals paid GitHub customers and gets the same workloads, they will start having similar issues! Stealing engineers who couldn't fix this at GitHub won't make much of a difference. GitHub is not just source control; everybody can set up Gitolite and have an amazingly configurable and secure Git setup. I did this 10 years ago, and CODEOWNERS, protected branches, and tags can't do 1% of what I had done before. Cursor is going all over the place as it can't compete with their model. So, this is a defeat, a pivot, not something glorious!
- bilalq
It wasn't that long ago that S3 didn't have read-after-write consistency. Crazy to see the explosion in cool stuff built on top after that change. This is a really well written article and definitely changes my perception about Cursor's capability to host a reliable Github replacement. But I could never bring myself to use it because of whose leadership they're under now. It's such a shame.
- wibr
https://cursor.com/en-US/blog/git-at-any-scale to avoid automatic translations (couldn't find a button to switch)
- zinodaur
Great read - the main takeaway I had was “Spokes has an insane replication protocol for a database (Git) with such low write rates”. Leader/follower + LSN Binlog, WAL, whatever - just not their distributed systems nightmare coordination system
- ljm
The problem is that Musk acquired Cursor and therefore Cursor has zero credibility.
I don't care about what they say because of who they sold out to