How a 512Mi OOM Killed Our Cluster's 'Load Everything' Assumption
State-oriented consistency: Why we stopped looking for one right answer
A distributed message broker kept OOM-killing pods under identical memory loads, revealing a design flaw: each node loaded the entire fleet's session state at boot. The fix wasn't tuning but reclassifying state by its consistency needs. The authors introduce 'State-Oriented Consistency': ask what happens if two nodes disagree, then pick the weakest guarantee that works. This reframe turned a memory bug into an architectural insight, showing that consistency is per-state, not per-system.
Consistency is not a property of a system. It is a property of individual pieces of state.
- fcravio
Author here. This came out of a production OOM in a clustered MQTT broker. The main lesson was that we had been treating different kinds of distributed state as though they all required the same consistency guarantee. Happy to discuss the incident, the ownership model, or the design trade-offs.