We Turned Off Pub/Sub and Nobody Noticed
At incident.io, our entire platform runs on event-driven architecture, with Google Cloud Pub/Sub as our sole message broker. But as we tightened our availability targets to 99.99%, Pub/Sub became a single point of failure. To fix this, we built a dynamic load balancer that splits traffic between Pub/Sub and NATS, with automatic failover. After months of work, we successfully turned off Pub/Sub entirely with zero customer impact. This post details our journey, from choosing NATS to implementing a fair scheduling algorithm.
Our message broker had become a single point of failure (SPOF), and this was at tension with our own 99.99% availability targets.
- rorymalcolm
Disclaimer: I work at the incident company
This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.
Never not scary, never not eventually fun, but type 2 fun for sure.
- shintoist
NATS is such a cool piece of technology that deserves more hype than it gets