Feature Flags: When They Help and When They Hurt
When Feature Flags Do and Don't Make Sense
Feature flags are powerful but come with real costs. Rajiv Prab shares his experience from Amazon and other teams, explaining when they shine—A/B testing, complex epics, and uncontrollable deployments—and when they become a liability. He warns that overusing flags adds complexity, creates tech debt, and can lead to bugs, citing the Knight Capital disaster as a cautionary tale. The key is to use them judiciously, not as a substitute for solid testing and rollback practices.
A flag that hasn’t been set to off in a year can be masking a major regression.
- paulryanrogers
Worked at a place with a literal settings table of 5K (non-default) records, it was many hundreds of possible settings.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
- stopping
I never found a great way to incorporate feature flags into my workflow without inducing significant mental churn managing 12-step rollouts over a dozen independent active flags. The changes I tend to make are sweeping, non-trivial refactors of base libraries with hundreds or possibly thousands of callers. These sorts of changes are exceptionally hard to flag (especially API changes), and it's stupidly easy for another developer to fat-finger a merge conflict resolution and drop one of my flag gates.
To this day I've never found any good guidelines for flagging changes like this without resorting to widespread file duplication, abuse of OOP, or an ad-hoc versioning system. It comes as no surprise that nobody in my organization was willing to do this sort of work.
- classictraffic
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
- MaulingMonkey
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
- conradludgate
I've been introducing feature flags into our component at work.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts