Linux's PCIe bridge sizing bug blocks four 32 GB GPU BARs behind one switch
Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux
A Broadcom PEX88096 switch feeding four AMD Radeon AI PRO R9700s and two RTX 3090s from a single x16 slot needs a full 32 GB Resizable BAR on each R9700 for ROCm's peer-to-peer mode. Linux 7.0's pbus_size_mem() sums child windows without leaving room for the 2 MB doorbell BARs, so only two cards fit. A 14-patch series from Ilpo Järvinen fixes the sizing, and an out-of-tree initramfs module pads each window to its alignment.
With the doorbell placed after each BAR, the way Linux 7.0 lays them out, the four windows span 224 GB. Two R9700s got their BARs, and the rest of the switch got nothing.
- dougcalobrisi
This is my post. Basically, I needed P2P between four R9700s for tensor parallel serving, which means a 32 GB BAR on each card. With all six GPUs in, the BIOS couldn't map them, and Linux sized the switch's window too small, so only two cards got their BARs. The module in the post works around that. With P2P off, prefill was about half as fast.
- zir_blazer
These kind of things should be fun to test in Coreboot. When Dasharo was released for the MSI PRO Z690-A DDR4 more than four years ago, I had issues with two Radeon 5600XT installed that was fixed after developer tuned some MMIO related parameters: https://github.com/Dasharo/dasharo-issues/issues/245
How many PCIe Switches you could chain and how many cards with absurd MMIO requeriments you can get working would be an interesing experiment.
In the last decade or so I have seen plenty of Firmware MMIO related issues. I remember certain SR-IOV supporting NICs (Usually Intel ones like i350 and other enterprise parts) not being able to reserve enough MMIO to create all the supported SR-IOV Virtual Functions in consumer Motherboards, in many cases you could get SR-IOV working but were limited to create like 4 VFs even if hardware supported 16 because that is what you could fit. This was essentially all a Firmware issue, not Hardware, but few people has the required skills to do any BIOS modding to fix rather convoluted things like MMIO allocation. Basically, you're trying to workaround Firmware issues elsewhere.
- bombela
The word or acronym BAR is mentioned 34 times but never defined.