The 'no healthy upstream' error that wasn't about CPU
When "no healthy upstream" isn't about the upstream you think
A search backend intermittently failed with 'no healthy upstream', but dashboards showed no CPU spikes or crashes. The inherited root-cause analysis blamed CPU throttling, yet evidence contradicted it. The real culprit was a datastore client with no timeout and aggressive retries, causing worker pool saturation and head-of-line blocking. The fix: add explicit timeouts and bound retries to prevent dependency blips from cascading into fleet-wide outages.
A CPU-throttled worker is ready to run but cannot get enough scheduler time. A worker blocked on I/O is off-CPU, waiting on the network while still occupying its worker slot.
- JohnMakin
> The instinct during an availability incident is to add headroom: raise CPU limits, increase the worker pool, add replicas. That can help with genuine capacity problems. Here, it would only give the retry loop more workers to occupy.
This is something during production issues I have a really difficult time sometimes communicating to peers. "Our services are timing out, we're seeing high latency, increase all resources!" is the knee jerk response, but sometimes, and even often, if the underlying cause of the degradation is something like, a database locking up, increasing workers and giving them more firepower might just make the situation even worse. It happens a lot more than you would think.
- ppedra
Wow, that`s very true...
> My disability is not a tragedy of biology. It is a failure of infrastructure.
Everybody is different. Tall, short, strong, week, two legs, one leg, short sighted or blind. So it makes a lot of sense to point (as a mistake) that our infrastructure favors one type over the others.
> the "Checkout" button is labeled "Button_Graphic_v2_Final"
This is a very good point.
- jtc331
Couldn’t even be bothered to disguise LLM generated content.
Painful to read.