CM4 hangs when performing large writes on PCI BARs mapped as normal memory #4928
Comments
|
The BCM2711 only supports up to 32-bit accesses to the Root Complex - that is a fundamental hardware limitation. Only device drivers that map as device memory will work. Closing as Can't Fix. |
Is a rather important fact but I failed to locate this information in any document published by the foundation. |
|
@mi-hol I don't think it's ever really been documented that thoroughly. As time progresses, and the community has been testing more and more cards and devices on the Pi, these 'features' are being uncovered. Until the CM4, only a handful of people on the planet ever tried any PCI Express devices other than the VL805 chip on a BCM2711 (afaik). It might be nice at some point to have more thorough documentation about what calls to avoid when working on drivers for the CM4 in particular, though. I've seen some interesting projects people are doing (or trying to do) with a CM4 where it's a decent fit, but some of the projects stall out when they run into these driver issues, and it's really a process of manual debugging. I've been documenting the rough edges I've hit so far and hope to summarize most of what I learned in an upcoming post/video though. At least that'll be something when people search for a concise list! |
@pelwell therefore these learnings should be added to RPis official documentation, just closing this issue without follow-up on side of documentation doesn't feel right |
|
From page 8 of the CM4 datasheet (https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf): |
|
@pelwell great to see it mentioned in the datasheet. Now from my view its a bit hidden there. |
|
If you care about it that strongly, submit a pull request: https://github.com/raspberrypi/documentation/pulls |
|
Does anyone know of this 32GB limit exists for the BCM2712 used on the Pi 5? |
Describe the bug
Writes to PCI BARs that have been mapped as normal memory with a size of more than ~256 bytes freeze the BCM2711.
Commonly a lot of drivers assume it is ok to ioremap PCI BARs as normal memory (ioremap_wc) in contrast to device memory. However performing large writes on such mappings seems to somehow freeze the BCM2711. I've tested this with the Raspberry Pi CM4, a carrier board and multiple PCIe devices (e1000e NIC, SM750 GPU, AMD RX570 GPU).
The issue does not occur when mapping the BAR as device memory. However this does of course come with additional restrictions over normal memory, most notably need for aligned access. This is unexpected in many places, especially if such memory is exposed to userspace via mmap (i.e. on framebuffer devices).
Other arm and arm64 platforms do not seem to have this issue (tested on Ampere Altra). Is this expected behaviour and a known limitation of the PCIe controller on the BCM2711?
Steps to reproduce the behaviour
lspci -nn)dmesg | grep BAR, any mem BAR >=1kB will be fine)insmod bar-write-test.ko vendor_id=0x8086 device_id=0x10d3 bar=1, replace vendor, device id and bar with the ones from your peripheral)Trying 2^9...as the last kernel log messageRepeat the same test, but insert the bar write test module with additional parameter
device_mem=1(insmod bar-write-test.ko vendor_id=0x8086 device_id=0x10d3 bar=1 device_mem=1). This maps the BAR as device memory and the test will complete successfully without hanging the entire system.The bug can also be reproduced with the sm750fb driver and a SM750 based graphics card. Those tend to be quite expensive and hard to come by though, thus I deem above reproducer more precise and accessible.
Device (s)
Raspberry Pi CM4
System
rpi-issue:
Firmware version: a26faf97e3bf76bcc23949d7cdab2f96f399a0c3 (clean) (release) (start)
Kernel version: 5.15.24-v8+
Has also been tested with Linux 5.10.92-v8+ before, behaviour is exactly the same
Logs
Log output was taken from serial with loglevel set to 9.
Log output when remapping as normal memory:
Log output when remapping as device memory:
Additional context
No response
The text was updated successfully, but these errors were encountered: