S2 ditched Route53 to run its own DNS server, then hid it behind Cloudflare
It's always DNS, so why are we running our own?

S2 needed unlimited, instantly resolvable DNS records for its basins, but Route53's API limit of about 5 UPSERTs per second made that impossible. So the team built a custom authoritative DNS server in Go, backed directly by Cloud Spanner, and put Cloudflare's DNS Firewall in front to absorb attacks. The system now handles thousands of basins with instant availability, though the delegation chain is a tangle of Cloudflare, GKE, and Route53.
We tried building in-app defenses for NXDOMAIN however, it is, in my opinion, the worst DDoS attack and nothing can really stop it.
- reticulates
I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.
- icedchai
How frequently are new DNS records being added / modified?
- MiscIdeaMaker99
Back in the day, I remember running our own DNS servers because we had customers who wanted to white-label our product with their own domains, and _lots_ of marketing departments didn't want to deal with their IT people, so we offered to host DNS for them. Of course, that's assuming they had IT people at all.
About half of the domains were registered by us on behalf of the customer, while the other half delegated a sub-domain to us. There was also a small minority who hosted all of the DNS themselves.
- bajard
My ISP kept intercepting, editing and blocking my DNS queries. I had to encrypt, obfuscate, self-host, re-route and randomize them to get rid of this harassment.
- whinvik
I am very curious how they decided to have most infra in GCP but only DNS on AWS Route 53. I would expect that one could have everything on just GCP no?
The other interesting thing that I am starting to think more about is that every part of the infra stack will get slowly rebuilt because of AI. More load, more granular permissions, more firewalls are some of the things I can think that will need to be built in.