Treat Your Development Pipeline as a Critical Production System
The development pipeline is a production system
Software teams often rush to fix customer-facing outages but ignore broken build systems or downed QA servers. I argue that any failure in the development pipeline, from GitHub Issues to CI/CD tools, stops value delivery just like a production crash. When developers cannot compile code or testers cannot run checks, the entire team halts. We must treat these internal failures with the same urgency and priority as external service outages to ensure continuous software production.
For the development team, the development pipeline is a production system.
- tetha
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
- hnlmorg
This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)
- wxw
In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common.
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
- donatj
> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
- dsjoerg
There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"