Why We Rebuilt Kafka on Kubernetes Before It Broke
What Running Kafka on VMs Taught Us About Systems Thinking
Moniepoint's Cloud Engineering team ran Kafka on VMs with Docker Compose. It worked, but every change meant SSH, restarts were risky, versions drifted, and monitoring was manual. So they proactively migrated to Strimzi on dedicated GKE clusters, using namespaces per pipeline, GitOps with ArgoCD, and a custom Kafka Connect image. The result: no more SSH, self-healing brokers, rolling updates, and a unified view. The lesson: manual operations consume the cognitive space needed for systems thinking.
You can't think in systems when you're drowning in operations.
- sgarland
> Our Kafka setup started the way most things do in a fast-moving engineering team: pragmatically.
> Every change required SSH and port-forwarding. Our Kafka instances had no URL. You accessed them via localhost.
That isn’t pragmatism, it’s lift-and-shifting your test environment into prod. You can in fact run stuff on VMs in a repeatable and sane way; it’s called Ansible + Packer (or any other similar combo you’d prefer).
- zulux
I've had to learn to change how I write so I'm not accused of being a bot. No dashes, no trite pairings. (Yes that was intentional)
It just dawned on me that I've had to do this before: When PC stuff became vogue: Black vs white. Cis Female vs women.
- MrBuddyCasino
> „It Wasn't a Crisis. It Was a Decision.“
closes tab