A 40ms Go GC pause caused by swap
A 40ms Go garbage collector pause caused by swap
I ran swap in production to absorb memory spikes, but it backfired: Go's garbage collector reads its metadata in a stop-the-world pause, and that metadata can end up in swap. On a Hetzner box with NVMe, the worst pause hit 40ms, with 39ms spent on 228 page faults inside GC bookkeeping. The median pause is only 51 microseconds, so this is 800 times worse. I also saw message building jump from 3-5ms to 105ms on NVMe and 903ms on network volume. Go 1.26's Green Tea GC doesn't change this.
Those 40ms seem harmless at first. But we are talking about a stop-the-world pause. Those 40ms mean everything has stopped - in Go's terminology, every P has stopped, so, for example, if a goroutine was waiting for I/O, during that pause the I/O might return and there would be no one to handle it.
- nasretdinov
Unfortunately that's somewhat expected - regardless of implementation, the sole fact that the GC needs to somehow walk the entire tree to trace still referenced sections makes swap highly impractical -- even if you had not had 40ms stop-the-world pauses the LRU cache used by swap would get thrown away at every GC.
I believe that's actually the same reason why Apple stopped using GC in their frameworks in favour of automatic reference counting.
- jacobgold
"It hurts when I do this"
"Stop doing that"
If you care about latency, disable swap. System wide or for the specific the cgroup.
- xavdid
Discord learned this back in 2020 and published a great blog post on it: https://discord.com/blog/why-discord-is-switching-from-go-to...
- pizlonator
Is there a reason why Go isn’t using on the fly GC, where there’s no STW at all?
- soltanov
A GC latency SLO should include operating-system memory pressure. Otherwise, a page-fault problem will look like a collector problem and lead to the wrong fix.
- Gabrys1
I started wondering if there could be swap-aware GC, like first make the required page swapped in (not that there's any obvious API for that...) and only then pause the world?
- imclaren
This. Memory bloat bugs are relatively easy to fix in Go, but sometimes it’s an adventure to remove swap bloat. And I think it’s worth removing all swap bloat!
- faangguyindia
I use Go where i'd use Node, Ruby or Python.
I use Rust where i need low latency.