PCI-E P2P support on 3090 GPU

[geohot]

I have 2 non NVLINKed 3090s in my workstation, but I can't get them to communicate directly. I'm curious why it's not supported. P2P is supported over NVLINK, but not PCI-E.

Is there actually hardware missing, or is it just disabled somewhere? I can't see why a DMA would care if it's going to CPU ram or another GPU.

Is it related to the P2P mailboxes and the BAR mapping?

[geohot]

If the mailboxes are not present in the hardware, could mapping the full GPU RAM on BAR1 work? Similar to how AMD did it. https://www.phoronix.com/news/AMD-Multi-GPU-Compute-P2P