Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data transfer over USB becomes unreliable after upgrading to v5.4.83-v71+ #4084

Open
TupelUK opened this issue Jan 20, 2021 · 12 comments
Open

Comments

@TupelUK
Copy link

TupelUK commented Jan 20, 2021

I think there is a bug in the latest pi4 Linux release, because data transfer over USB becomes unreliable after upgrading from Linux v4.19-v71+ to v5.4.83-v71+ on raspberry pi4.

We are using the USB C port as a RNDIS gadget to get TCP data to a PC. I found the minimum python code required to reproduce the problem, it sets up a socketserver on the pi4, and the client on the PC repeatedly asks for 10MBs data. In Linux v4.19-v71+ it is robust, and I can do the transfer > 5000 times (expected behaviour). In v5.4.83-v71+ exactly the same code works 50 to 500 times then the link starts to break down, I get time out errors (even with time out set to 10 seconds), and frequently after that the link freezes entirely and requires the USB to be disconnected and reconnected for it to return. It even takes down an SSH connection to the pi which is separate software (but connecting over the same USB cable).

I'm unable to use v4.19-v71+ from now because it is not compatible with the latest pi4 hardware, so I hope someone can help - either with a bug fix or workaround suggestions.

To reproduce:
Test code I am using is here:
USBdataTest.zip

Set up the pi4 as a usb gadget:
To /boot/config.txt add:
dtoverlay=dwc2
To /boot/cmdline.txt add:
modules-load=dwc2,g_ether
and use the /etc/network/interfaces file provided.
On the pi4 run:
python3 lanTestPi2.py
On a windows PC run:
python3 lanClientTest.py
It can be reproduced with or without an external PSU for the pi.

Other information:
PC OS: Windows 10
Pi4 boot loader: Sep 3 2020
Python version: 3.7

  • Which model of Raspberry Pi? Pi4
  • Which OS and version? 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
  • Which firmware version?Jan 8 2021 14:32:15 version 194a85abd768c7334bbadc3f1911c10a7d18ed14 (clean) (release) (start_x)
  • Which kernel version? Linux TupelPi1 5.4.83-v7l+ BCM2708 I²C driver stops working sometimes #1379 SMP Mon Dec 14 13:11:54 GMT 2020 armv7l GNU/Linux
    I upgraded by using:
    sudo apt update
    sudo apt full-upgrade

Logs:
Time out error (from PC recv command).
No error from pi4, but PC can no longer connect to it.

Please let me know if there is any other information I can provide, or if I can do any tests for you to narrow down the problem.

@TupelUK
Copy link
Author

TupelUK commented Jan 21, 2021

I have managed to narrow down the issue to the particular github commit that causes it.

If I do:
sudo rpi-update ab2197b00eb5ba600556c37ac79b9e05a41e767d
(thats the Oct 8, 2020 commit of the Hexxeh/rpi-firmware repo)
I get expected network behaviour.

If I do:
sudo rpi-update 98d907955a74ba1833f9c31b1fe11b237f8ba599
(thats the Oct 15, 2020 commit)
The network freezing happens.

The (for me) troublesome commit say something about "Fix USB init for 2835 bootcode.bin".

This gives a solution to our issue for now, because the functional Oct 8, 2020 commit is new enough to work on the new pi4 hardware, and old enough not to have the network freeze issue...but I would be surprised if it is not affecting others.

@TupelUK
Copy link
Author

TupelUK commented Jan 21, 2021

FYI: the issue is also present in the latest commit I tried: Jan 18, 2021: commit number: 246f5ead8d2d017fd2734d79e1f9a606fb65b72c

@popcornmix
Copy link
Collaborator

@timg236 does "Fix USB init for 2835 bootcode.bin" affect start.elf and could it have an impact on later USB use?

@timg236
Copy link
Contributor

timg236 commented Jan 21, 2021

Not as far as I can tell. This could would only be used if rpiboot (usb device boot) is used. The change was to the ensure that soft disconnect was still called between bootcode.bin and start.elf on a Pi3 in USb mode.

@timg236
Copy link
Contributor

timg236 commented Jan 21, 2021

fw-github-linux-4084.zip

There's a separate commit around then which might be squashed. Please could you try 'sudo rpi-update 98d907955a74ba1833f9c31b1fe11b237f8ba599' and apply the attached experimental (untested fix) which disables some USB PHY reset code.

@TupelUK
Copy link
Author

TupelUK commented Jan 22, 2021

I tested the files, but unfortunately it did not fix the issue.

@timg236
Copy link
Contributor

timg236 commented Jan 22, 2021

I've run the attached python scripts plus some other SSH and HTTP download soak tests with the latest APT FW release with a CM4 + Pi4. No errors observed. There's another theoretical change attached

github-linux-4084-v2.zip

@timg236
Copy link
Contributor

timg236 commented Jan 22, 2021

I managed to get a failure after 'pursuading' RNDIS to work on my Windows 10 machine. It's possible that this won't fail on CM4 + IO board.

Attached is a possible fix candidate which passed 1000 iterations a couple of times so I'll leave it running over the weekend. I think v2.zip won't fail either but this more targetted

github-linux-4084-v5.zip

@TupelUK
Copy link
Author

TupelUK commented Jan 25, 2021

Thanks so much for the rapid response. I've run some more tests:

On clean install:

So the 4084-v5.zip files fix the problem on a fresh install, which is great.

I realised that the install I was testing with was quite modified and it seems this enhances the problem. The 4084-v5.zip does not fix it, even though it does work on the clean install.

On modified install:

  • linux commit 98d..., 4084-v5.zip files connection frozen completely after 250 tests.
  • linux old commit 50b..., 4084-v5.zip files pass 2x 1000 tests - no fails.

I listed the changes that seem most relevant to the usb:

So for now (since I need my modifications), I'm using the old 50b... commit as a solution.

I realise that this could be becoming an ever more challenging issue to debug, but if you think it could be important I'm happy to help if I can.

@timg236
Copy link
Contributor

timg236 commented Jan 25, 2021

Thanks for the test results. V5 restores a single PHY register setup operation which shouldn't be relevant on BCM2711, unfortunately, that code is pretty ancient so getting a definitive answer may take a while.
Since v5 improves things I'll try to get it merged as a partial fix until the Linux side is understood. It's also unusual that it takes so long to fail when this sort of problem normally causes enumeration or device configuration to fail.
I'll post an update when I know more.

popcornmix added a commit to raspberrypi/firmware that referenced this issue Jan 28, 2021
firmware: usb: Fix dropouts with USB ethernet gadget
See: raspberrypi/linux#4084

firmware: imx477: Allow long exposures for the binned modes.
See: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=297521
popcornmix added a commit to Hexxeh/rpi-firmware that referenced this issue Jan 28, 2021
firmware: usb: Fix dropouts with USB ethernet gadget
See: raspberrypi/linux#4084

firmware: imx477: Allow long exposures for the binned modes.
See: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=297521
@timg236
Copy link
Contributor

timg236 commented Jan 28, 2021

The firmware fix is now available in the latest rpi-update kernel. I've been testing with the 5.10 kernel which is now the default rpi-update kernel. The APT packages are switching to 5.10 very soon.

@raspberrypi raspberrypi deleted a comment from SpaceRat42 Jan 28, 2021
@timg236
Copy link
Contributor

timg236 commented Jan 28, 2021

Deleted off topic comment

mkreisl added a commit to xbianonpi/xbian-package-firmware that referenced this issue Dec 3, 2021
- firmware: isp: Fix handling of different YUV colour spaces

- firmware: poe_hat: Actually close the I2C handle

- firmware: platform: Define DVFS modes and change default to be fixed AVS voltage

- firmware: arm_loader: Auto-select 64-bit for kernel8.img
  See: #1193

- firmware: hdmi: Throttle auto-i2c register writes to avoid PWM audio underrun

- firmware: platform: Define DVFS modes and change default to be fixed AVS voltage

- firmware: arm_loader: Auto-select 64-bit for kernel8.img
  See: #1193

- firmware: hdmi: Throttle auto-i2c register writes to avoid PWM audio underrun

- firmware: video_decode lockup handling

- firmware: isp: Initialise extras to avoid vpitch being random

- firmware: usb: Fix dropouts with USB ethernet gadget
  See: raspberrypi/linux#4084

- firmware: imx477: Allow long exposures for the binned modes.
  See: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=297521

- firmware: arm_dispmanx: Use ALPHA_MIX flag
  See: https://www.raspberrypi.org/forums/viewtopic.php?t=300769

- firmware: power: Refactor the interface to the PMICs

- firmware: platform: vl805: Get BAR2 address from PCIe BAR2 registers

- firmware: arm_loader: Return all borrowed DMA channels
  See: #1541

- firmware: hdmi_2711: Rework I2C driver to NOT use the AUTO-I2C block

- firmware: gencmd: Allow groups of clocks/plls to be read together

- firmware: power: Fix DA9090 under-voltage detection

- firmware: NVME boot support

- firmware: brfs: Fix USB bulk-read in start.elf
  See: Hexxeh/rpi-firmware#258

- firmware: hdmi_2711_i2c: Correct handling of start/stop codes for long read
  See: #1548

- firmware: video_decode: For VC1/WMV with no signalled header bytes, use start of 1st buffer
  See: raspberrypi/linux#4113

- firmware: vl805: Remove redundant log statement and fix warning

- firmware: power: Fix DA9090 ADC1 register definition

- firmware: arm_loader: Only report clocks arm has set, not siblings

- firmware: arm_loader: Don't report clocks set as turbo side effect of arm clock

- firmware: arm_loader: 2711: gpu clocks are not dependant

- firmware: platform: Need to clear cached versions of get_max_clock_internal vars

- firmware: Move core to PLLA and support accurate clk108
  See: xbmc/xbmc#19263

- firmware: board_info: Separate memory size from OTP field encoding

- firmware: power: Swap DA9090 ADC assignments to match XR77004

- firmware: board-info: Fix memsize on 3B+

- firmware: vcfw/power: Add a new latch for power_pad_control
  See: #1552

- firmware: arm_loader: kernel_old=1 should force kernel_address=0
  See: #1561

- firmware: scalerlib: Fix offset applied to x coordinate of YUV10COL image
  See: https://forum.kodi.tv/showthread.php?tid=361164&pid=3024654#pid3024654

- firmware: isp: Ensure the VRF is locked when setting up video colour denoise
  See: raspberrypi/rpicam-apps#19

- firmware: isp: Remove custom EV mappings from camera tunings

- firmware: Add support for board-type=0xXX conditional filters in bootloader, bootcode and firmware

- firmware: Two UART1 patches
  See: #1566

- firmware: Pi400: Reduce MII clock freq when probing ethernet PHY

- firmware: platform: Remove build-time constant for MICROVOLTS_PER_PIP

- firmware: dt-blob.dts: Correct HDMI HPD and EMMC_ENABLE for CM4
  See: https://www.raspberrypi.org/forums/viewtopic.php?f=29&p=1858516

- firmware: vcfw/hdmi: CUSTOM modes used for FKMS didn't set RGB quant range correctly
  See: #1580

- firmware: PoE+ HAT support
  See: raspberrypi/linux#4367

- firmware: arm_loader: Use Pi4 bootloader MAC_ADDRESS if set

- firmware: platform: Apply ARM thermal throttling rules on BCM2711

- firmware: bcm_host: Recognise all Pi 4 variants, add BCM2711
  See: raspberrypi/userland#695

- firmware: video_decode: Use the ISP instead of vc_image_convert

- firmware: hdmi-2711: Wait for HDMI hardware scheduler to activate in HDMI mode

- arm_loader: Add message to release firmware framebuffer

- firmware: arm_loader: Add rng-seed DT property
  See: #1595

- firmware: isp: Set the YUV420/YVU420 format stride to 64 byte

- firmware: Revert: video_decode: Use the ISP instead of vc_image_convert

- firmware: cec: Avoid sending messages with kms
  See: raspberrypi/linux#4460

- firmware: hdmi_cec: Remove TX/RX SW_INIT on power_on
  See: Hexxeh/rpi-firmware#267
  See: https://www.raspberrypi.org/forums/viewtopic.php?p=1895082#p1895082

- firmware: arm_dt: Limit CMA to 256MB if total_mem < 2GB or gpu_mem > 256MB
  See: #1603

- firmware: video_decode: Use the ISP instead of vc_image_convert

- firmware: video_decode: Correct support for YVU formats using ISP

- firmware: firmware: Disable VLL loading from file system
  See: #1605

- firmware: arm_loader: Make most arm clock requests required
  See: #1598

- firmware: arm_loader: Consider required flags from GET_CLOCK_RATE
  See: #1598

- firmware: arm_dt: Load overlays for detected cameras

- firmware: Make more use of the user-warnings DT property

- firmware: hdmi_2711: Use HDMI block REPEAT_PIXEL instead of PV
  See: https://forum.libreelec.tv/thread/24415-le-10-beta-for-i4-force-hdmi-resolution

- firmware: DSI display autodetection for kms

- firmware: arm_loader: Allow hvs interrupt during SET_NOTIFY_DISPLAY_DONE

- firmware: arm_display: Allow null buffer in successful call
  See: raspberrypi/linux#4540

- firmware: video_decode: Ensure all buffers are flushed before port disable completes

- firmware: filesystem: sdcard: Fix Hybrid GPT partitions
  See: #1465

- firmware: tvservice: Add check to warn when running with kms

- firmware: arm_loader: Allow non-optional reads of current clock
  See: #1619

- firmware: dispmanx: Demote null eptr from vcos_verify to no warning
  See: raspberrypi/linux#4592

- firmware: filesystem: sdcard: Probe FAT type in GPT ESD partitions

- firmware: clock-2711: Limit PLLB VCO frequency to the high range

- firmware: arm_dt: Export the boot-mode, partition and usb state via device-tree
  See: #1621

- firmware: video_decode: i/p port enable/disable without o/p active could stall
  See: RPi-Distro/vlc#48
  See: Hexxeh/rpi-firmware#272
  See: #1637

- firmware: userland: Reduce debug_sym error messages
  See: https://forums.raspberrypi.com/viewtopic.php?f=98&t=322238

- firmware: arm_dt: Increase maximum line length to 98
  See: raspberrypi/linux#4638

- firmware: arm_loader: Allow VEC clock to be controlled by arm

- firmware: platform: Remove licence on VP6, VP8, Theora, and FLAC
  See: raspberrypi/linux#4661

- firmware: ISP: Fix magenta colour in right hand image of stereo pair
  See: https://forums.raspberrypi.com/viewtopic.php?t=321089

- firmware: platform: Fix incorrect turbo voltage scaling on Pi0
  See: raspberrypi/documentation#2255

- firmware: platform: Declare CM4's SIO_1V8_SEL and SD_PWR_ON
  See: raspberrypi/Raspberry-Pi-OS-64bit#188

- firmware: hello_fft: Update outdated link to V3D spec

- firmware: hello_fft: Remove unused function declaration
  See: #1645
  See: raspberrypi/userland#710

- firmware: dtoverlay: Rebase aliases in overlays like labels

- firmware: isp: Set core/vpu min clock to 320Mhz during ISP operation

- firmware: arm_loader: Enable watchdog early if wanted
  See: #1651

- firmware: board_info: Add upstream dtb names for cm1 & 3

- firmware: board_info: Add upstream dtb name for cm4
  See: #1660

- firmware: platform: Allow users to disable camera boot HMAC check
  See: #1657

- firmware: clock: 2711: Fix potential API issue in 2711 VCO setup

- firmware: arm_loader: Enable USB MSD boot mode on Zero 2 W

- firmware: isp: Fix Rec.709 colour space problems
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants