Closing the Xen dom0 I/O Gap with NUMA Awareness
Making Xen's dom0 I/O path NUMA aware
I fixed a critical gap where dom0 memory was skewed toward low-address nodes on multi-socket hosts. By implementing proportional memory allocation and synthesizing accurate SRAT, SLIT, and x2APIC topology data, dom0 now sees the real hardware layout. This ensures that even unsandboxed containers running directly in dom0 benefit from proper NUMA balancing, preventing performance regressions when adopting Xen.
A non-Edera pod could end up slower on an Edera host than it had been on the same box as plain Linux, purely because turning the machine into a Xen dom0 hid the topology its workloads used to see.
- unixhero
Why is Xen a thing when we have great and free KVM virtualization? Honest and dumb question.
- twoodfin
This article is a great example of the limits of LLM technical writing.
Say what you will about—or try to prompt out—their rhetorical flourishes, they can link together a series of background, observations, implications, …
But man, they are hopeless at crafting a high-level narrative. It’s one big hedge maze of a garden path:
This part makes good on that - and along the way it covers two things we did not plan to fix. One we hit before the real work could start: dom0's own memory placement was so skewed that synthesising a topology for dom0 without fixing it first would have been an exercise in lying to a kernel that had no memory to back the lie with. That one is a prerequisite, not a detour - everything else in this part is built on top of it. The other we hit after the work was nominally done: a memory-placement bug in our own toolstack that had been silently broken for as long as multi-vnode vNUMA had existed in our tree, and only ever surfaced under a memory-bandwidth benchmark. That one is the genuine detour, and it waits until the end.
Let's start with the prerequisite.
Where are we going here and why?!