How to ship a database every day
turbopuffer deploys dozens of database upgrades daily across 100+ clusters, including BYOC clusters it can't directly access. The key is a local Kubernetes controller that drives operations via a custom resource, while a central control plane (API + MySQL) handles work distribution and status tracking. This design allows engineers to ship fast without DevOps, using a keyboard-driven UI inspired by Linear.
We don't want different control planes for BYOC and SaaS, so we have to design for the lowest common denominator.
- dwedge
This might seem like bait but I assure you it isn't - I reach for MySQL by default over postgres - but why did you choose MySQL for the state database? In terms of HA and synchronous clustering it's a good solution but unless I'm misreading it seems like you only use it for a key value store and for an append only log, neither of which are necessarily the problems you think "MySQL" for.
Given what seems to be a pretty fashionable stack (kubernetes, using etcd directly, cloud first, IaC and custom database technology) I'm interested what a company like that reaches for MySQL. Was the MySQL cluster already deployed for other reasons?
- robszumski
I was part of the team that proposed this manner of operation (Kubernetes Operator)[1]/controller and it's immensely cool to see a competent team make what seems like an incredible one that really scales the business.
I had always assumed databases would be the silo that had the most benefits for an Operator. There are a few good Postgres Operators, but not many outside of that which actually do the day 2 operations.
- hemc4
Interesting read. A simpler way to manage the data plane.
Curious to understand if you also use OTEL layer to understand what is happening in your BYOC cluster ?