PlanetScale's 50 GB/s backup system: how sharded Postgres backups scale to petabytes
Massively Parallel Postgres Backups
PlanetScale's sharded Postgres databases back up at over 50 GB/s, enabling petabyte-scale backups in hours. The system spins up temporary EC2 instances per shard, restores the previous backup from S3, replays WAL from a mix of S3 and the primary, and saves the new backup. This parallelism makes backup speed scale with shard count: a 32 TB database on 8 shards backs up in ~2.8 hours, while 100 TB on 100 shards matches the speed of 1 TB on one shard. Backups also power everyday operations like resizing and node replacement.
100 terabytes on 100 shards backs up at ~the same speed as 1 terabyte on a single shard.
- bddicken
What I emphasized in this article (author here) is scaling postgres backups. This works because there's already rock-solid systems built into postgres + surrounding tooling to build from.
The broader takeaway is the principle of, "how do I take something that doesn't scale on its own and make it so?" This applies to backups, compute, storage layers, proxies. It's why Neki and Vitess are so powerful for everything from small 1GB databases to petabytes.
Hanging around to answer questions, too :)
- Onavo
Interesting, last I checked PlanetScale still doesn't have in place Postgres version updates.