Uber's Fix for Retry Storms: Let Services Own Their Errors
How Uber Protects Against Retry Storms
Retry storms can turn a single service outage into a stack-wide incident, and traditional retry budgets aren't enough. Uber's solution: a context-aware mechanism that establishes error ownership. By correlating inbound and outbound failures, services claim or unclaim errors, so retries only happen where they can help. This prevents amplification across deep dependency chains while preserving at-least-once retries.
During periods of high error rates from a service, the errors are less likely to be randomized, and wouldn't benefit from a higher number of retries, and instead might be responsible for further degradation.
- prologic
So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
- Scoundreller
Meanwhile Google keeps giving me “please wait, do not reload page” walls, so I ctrl-r as rapidly as possible. Or is that the human test and response?
- maxchisto
I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
- aftbit
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
- UltraSane
This feels like trying to reinvent Fibre Channel's flow control mechanism.