Pi4 Arm64 USB Device descriptor errors #3093
Comments
|
It's caused by d5dc848. |
|
It sounds like the DMA bounce buffer support needed by the PCIe block is either not hooked up or not working. |
|
How does that come into play given that we limit now the dma to 1G? |
|
My understanding (and this whole area is a bit murky and complicated) is that the DMA zone limits where memory for dma_*_alloc is placed. PCIe is a bus master, so it is effectively also a DMA controller, but it can only get to the bottom 3GB of RAM. If there is data in the top 1GB to be read from our written to USB then something has to arrange to copy to or from an intermediate buffer before or after the transfer. This is a common requirement, and yet bounce buffers don't seem to have a common implementation - individual subsystems seem to do it differently. |
|
That would explain it indeed. I found the implementation of the bounce buffers for the pcie controller and saw that the threshold is set to 3G. This basically means that we need to set this to 1G as well, right? At least that is how it sounds to me. But I'm confused how come this works on 32bit as we similarly only limit the zone and the driver bounce threshold default to the same 3G. There is something I still miss in this story. |
|
[ Again - this is just what I think is going on, I can't guarantee it is correct. ] Think about what happens when an application needs to send some data to USB: a user-space pointer will get passed to a system call. At this point the driver or framework has a choice - whether to copy the data into an intermediate buffer for transmission or whether to attempt to use it in-place. The DMA Zone helps with the first case because it can guarantee that the intermediate buffer is in a DMA-safe location. It doesn't help (directly) with the second case because those user pages could be anywhere in RAM, so the PCIe driver hooks into the DMA methods for the USB device and (transparently to the user) performs a copy at the point the buffer is mapped for DMA, returning the address of the bounce buffer rather than the user pages. |
|
That was my understanding as well. Thanks for clarifying @pelwell . So in this case we need to make sure the threshold is indeed 1G. We need to allocate bounce buffers for everything that goes beyond this limit. The first 1G is marked as DMA safe-zone so everything else needs to go through bounce buffers. |
|
That's still not correct. For PCIe (which means for XHCI USB on the Pi 4), any memory within the first 3GB would be safe to use as a bounce buffer for anything in the last 1GB on a 4GB Pi4. PCIe is a DMA controller, it doesn't need to use the system DMA controller, so the 1GB limit does not apply. However there seems to be only one global DMA_ZONE configuration, so it must be set to the largest value safe for all DMA controllers. |
|
But that exactly what we do. The DMA_ZONE configuration applies to all DMA controllers (at least it should) and we set it to the first 1GB. The system controller can address that and also the PCIe should work with that limit. |
|
What I am saying is that a 1GB limit is unnecessarily pessimistic for PCIe as a bounce buffer threshold. Yes it's fine to allocate bounce buffers from memory below 1GB for PCIe as well, but don't require the use of bounce buffers for USB/PCIe accesses to RAM above 1GB - the threshold should be 3GB, or performance will suffer. |
|
Got it now. It makes sense. Thanks for expanding it. |
|
@pelwell I think I "found" the problem. The bounce buffers are only compiled when CONFIG_ARM is enabled. They don't compile on arm64. |
|
I have been bashing my head for a while now and it seems to not be as easy as I thought. The current implementation uses a lot of |
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
|
@agherzan Good job! Thanks! |
|
Step by step. |
|
@agherzan @pelwell This is the V5 of brcm pcie controller patches, there is mention of ARM64 part, maybe help? |
|
It's always good to see the approach others have taken, but I think that the multiple buses of BCM2711 and the different memory maps they see makes the per-device custom DMA ops a better technique. |
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
|
@pelwell Would it be a better approach to limit the ram by setting the gpu mem to 1024? Do you foresee any issues with that? |
|
The GPU can only address a single GB of RAM. 2711 has a register that controls which GB of RAM it uses, but a) it's only ever been the first GB so changing that will break everything, and b) persuading Linux to limit DMA allocations to a GB that isn't the first sounds like a new version of the problems we are trying to solve, so I think we should just pretend for now that the mapping is fixed. Setting GPU mem to 1024 (if it worked, and I don't think it would) would reserve the first GB (call it GB0) for the VPU. You now have an ARM complex that can't use address 0x000xxxxx, where the kernel likes to live, and that only has 2GB of usable RAM (GB1 and GB2) - GB3 is still inaccessible due to the PCIe wrapper problem we're trying to work around. Does that sound like progress? |
|
In summary, |
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
|
I've created a trivial patch to add bounce buffers on ARM64 for pcie-bcrmstb. I'm sure there is a much better way to do that, but this seem to be working for me.
|
I tried this patch, it works for me! |
|
Please submit this patch as a Pull Request - complete with a Signed-off-by: line - to make it easier for people to review. |
|
Here is a PR: #3144 |
How does one apply/install this .patch? |
|
This is fixed in apt firmware. |
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
The PCI controller on the Raspberry Pi 4 acts as a DMA controller and can only address the first 3Gb[1] of the memory space. On 32bit this was addressed by implementing bouncing buffers[2] but this currently has no support for arm64. In order to have working USB, as a workaround, we limit the ram to 3G. Fixes #446 [1] raspberrypi/linux#3093 [2] https://github.com/raspberrypi/linux/blob/rpi-4.19.y/drivers/pci/controller/pcie-brcmstb-bounce.c Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Getting USB device descriptor errors on boot of arm64 kernel using rpi-4.19.y branch.
All usb ports unusable and unable to detect devices.
Below is the dmesg log.
The text was updated successfully, but these errors were encountered: