A Server Lost Power at 00:32. We Found Out at 08:18

A storage server at Danube Data lost power at 00:32 UTC on 17 August 2026, silently taking down object storage, the container registry, and serverless deployments for over eight hours. The status page detected the incident within three minutes, but no one saw it until an engineer happened to check at 08:18. The postmortem reveals why one machine caused a total outage: erasure-coded chunks were spread across disks without ensuring they were on different machines, and a few kilobytes of bookkeeping records were stored with less redundancy than the hundreds of terabytes of customer data they described. The team has since added a phone-call pager and a watchdog that can power-cycle a dead server, but the storage layout and capacity headroom remain unfixed.

Monitoring that nobody is woken by is a record, not an alarm.

More from this day

2026-08-24