Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dwc_otg rpi3b+ 4.14.52-v7 usb/serial dropping data #2687

Closed
edwar64896 opened this issue Sep 19, 2018 · 24 comments
Closed

dwc_otg rpi3b+ 4.14.52-v7 usb/serial dropping data #2687

edwar64896 opened this issue Sep 19, 2018 · 24 comments
Labels
Close within 30 days Issue will be closed within 30 days unless requested to stay open Waiting for internal comment Waiting for comment from a member of the Raspberry Pi engineering team

Comments

@edwar64896
Copy link

edwar64896 commented Sep 19, 2018

I am interfacing with a device which exposes a serial interface over USB. I am regularly seeing data being "lost" or "dropped" in the kernel/driver layer. Attempts to use some of the dwc_otg configuration parameters are yet to be fruitful.

When I get a chance I'll post lsusb and dmesg dumps from the box in question. the serial device is a cp210x chip. The code is attempting to read a sequence of 17 bytes, however intermittently only 16 bytes are being received for the message string in question. This causes the high-level code to fail a checksum check. The same codebase has been used on a x86 machine with no issue. The USB layer in the RPI just seems to be randomly discarding bytes.
dmesg.txt

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 12M
        |__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/3p, 12M
            |__ Port 1: Dev 9, If 0, Class=Vendor Specific Class, Driver=lan78xx, 12M
        |__ Port 3: Dev 8, If 0, Class=Hub, Driver=hub/4p, 12M
            |__ Port 1: Dev 10, If 0, Class=Vendor Specific Class, Driver=cp210x, 12M

lsusb.v.txt

@malacalypse
Copy link

malacalypse commented Sep 25, 2018

I'm having a potentially related issue with USB MIDI. I get dropped events resulting in stuck notes or missed control messages. This is a low speed device. It does not seem to matter which port it is plugged into.

I'm running 4.14.70-v7+

[Tue 25 Sep 2018 20:38:05 GMT]$ lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/3p, 480M
            |__ Port 1: Dev 8, If 0, Class=Vendor Specific Class, Driver=lan78xx, 480M
        |__ Port 2: Dev 17, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 2: Dev 17, If 1, Class=Audio, Driver=snd-usb-audio, 12M
[Tue 25 Sep 2018 20:40:05 GMT]$ lsusb -v

Bus 001 Device 017: ID 1c75:0288
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x1c75
  idProduct          0x0288
  bcdDevice            2.00
  iManufacturer           1
  iProduct                2
  iSerial                 3
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          101
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      1 Control Device
      bInterfaceProtocol      0
      iInterface              0
      AudioControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdADC               1.00
        wTotalLength            9
        bInCollection           1
        baInterfaceNr( 0)       1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         1 Audio
      bInterfaceSubClass      3 MIDI Streaming
      bInterfaceProtocol      0
      iInterface              0
      MIDIStreaming Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdADC               1.00
        wTotalLength           65
      MIDIStreaming Interface Descriptor:
        bLength                 6
        bDescriptorType        36
        bDescriptorSubtype      2 (MIDI_IN_JACK)
        bJackType               1 Embedded
        bJackID                 1
        iJack                   0
      MIDIStreaming Interface Descriptor:
        bLength                 6
        bDescriptorType        36
        bDescriptorSubtype      2 (MIDI_IN_JACK)
        bJackType               2 External
        bJackID                 2
        iJack                   0
      MIDIStreaming Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (MIDI_OUT_JACK)
        bJackType               1 Embedded
        bJackID                 3
        bNrInputPins            1
        baSourceID( 0)          2
        BaSourcePin( 0)         1
        iJack                   0
      MIDIStreaming Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (MIDI_OUT_JACK)
        bJackType               2 External
        bJackID                 4
        bNrInputPins            1
        baSourceID( 0)          1
        BaSourcePin( 0)         1
        iJack                   0
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
        bRefresh                0
        bSynchAddress           0
        MIDIStreaming Endpoint Descriptor:
          bLength                 5
          bDescriptorType        37
          bDescriptorSubtype      1 (GENERAL)
          bNumEmbMIDIJack         1
          baAssocJackID( 0)       1
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
        bRefresh                0
        bSynchAddress           0
        MIDIStreaming Endpoint Descriptor:
          bLength                 5
          bDescriptorType        37
          bDescriptorSubtype      1 (GENERAL)
          bNumEmbMIDIJack         1
          baAssocJackID( 0)       3

Bus 001 Device 008: ID 0424:7800 Standard Microsystems Corp.
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass          255 Vendor Specific Class
  bDeviceSubClass         0
  bDeviceProtocol       255
  bMaxPacketSize0        64
  idVendor           0x0424 Standard Microsystems Corp.
  idProduct          0x7800
  bcdDevice            3.00
  iManufacturer           0
  iProduct                0
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           39
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol    255
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval               4

Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         2 TT per port
  bMaxPacketSize0        64
  idVendor           0x0424 Standard Microsystems Corp.
  idProduct          0x2514 USB 2.0 Hub
  bcdDevice            b.b3
  iManufacturer           0
  iProduct                0
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           41
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      1 Single TT
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      2 TT per port
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12

Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         2 TT per port
  bMaxPacketSize0        64
  idVendor           0x0424 Standard Microsystems Corp.
  idProduct          0x2514 USB 2.0 Hub
  bcdDevice            b.b3
  iManufacturer           0
  iProduct                0
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           41
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      1 Single TT
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      2 TT per port
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            4.14
  iManufacturer           3
  iProduct                2
  iSerial                 1
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12

Dmesg contains nothing interesting when packets are dropped, but the device info on connect is:

[69248.133919] usb 1-1.2: new full-speed USB device number 17 using dwc_otg
[69248.267433] usb 1-1.2: New USB device found, idVendor=1c75, idProduct=0288
[69248.267440] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[69248.267443] usb 1-1.2: Product: Arturia KeyStep 32
[69248.267447] usb 1-1.2: Manufacturer: Arturia
[69248.267451] usb 1-1.2: SerialNumber: 00000000001A

@edwar64896
Copy link
Author

If any admins have suggestions as to how I can further debug this, I would be more than happy to assist.

@P33M
Copy link
Contributor

P33M commented Sep 28, 2018

Can you add the parameter dwc_otg.nak_holdoff=1 in /boot/config.txt cmdline.txt and retry? If there is negligible improvement, then try with nak_holdoff=0.

@malacalypse
Copy link

I tried it with both configurations. Stuck events persisted in both cases, did not observe any difference in frequency or severity versus no nak_holdoff value set.

@malacalypse
Copy link

But, did you mean /boot/config.txt or /boot/cmdline.txt ? I put this line in /boot/config.txt as requested. But the other threads for USB diagnostics are referring to /boot/cmdline.txt for other dwc_otg variables.

@6by9
Copy link
Contributor

6by9 commented Sep 29, 2018

Typo. It needs to go in /boot/cmdline.txt.
You can check the command line on the running system with cat /proc/cmdline

@malacalypse
Copy link

Ok I’ll rerun these tests with that in cmdline.txt shortly.

@malacalypse
Copy link

Good news: nak_holdoff=1 improved it significantly, but still a very few dropped events. holdoff=0 seems to have fixed it entirely, although I only gave it about 5 min of good data. That said, it's a clear and marked improvement.

Does this help you figure out where the problem is, and is it possible to fix it so we can effectively guarantee that we'll never lose an event using these low speed devices?

Is there any negative side-effect of keeping nak_holdoff=0 in the meantime?

Thank you!

@edwar64896
Copy link
Author

Unfortunately neither setting of nak_holdoff has made a noticeable difference to my situation. Still seeing sequences of 16 byte messages returning.

@edwar64896
Copy link
Author

##2197

@malacalypse
Copy link

malacalypse commented Oct 1, 2018

@edwar64896 just checking you put that in /boot/cmdline.txt not /boot/config.txt right? And of course you did a reboot after? I had no differences until I put it in the former, not the latter.

@P33M
Copy link
Contributor

P33M commented Oct 2, 2018

@malacalypse What baudrate is in use with your device?

You run into this issue if you fill the device's internal receive FIFO within the 1ms hold-off window, so it's a function of device FIFO size and incoming baudrate.

@malacalypse
Copy link

@P33M I'm not using a serial device - just class-compliant MIDI and audio stuff.

@pelwell
Copy link
Contributor

pelwell commented Oct 2, 2018

MIDI is 31250 baud.

@malacalypse
Copy link

malacalypse commented Oct 2, 2018

Not over USB, there are no limits. That's only for DIN serial MIDI.

https://www.usb.org/sites/default/files/midi10.pdf : "The USB transfers MIDI data at rate hundreds of times faster than the original MIDI 1.0 hardware specification." (section 2)

USB MIDI is true USB, not USB->Serial. Devices consuming the MIDI commands MAY implement a serial converter with the downstream data but that will not be visible to nor operating on the USB bus directly.

@P33M
Copy link
Contributor

P33M commented Oct 5, 2018

Ah it's an Arturia Keystep device. Previous testing with these devices has shown that they do not have any internal buffering past the 64-byte maximum packet size - so any data generated by key events that exceeds 64 bytes in size since the last time the endpoint was polled results in data loss.

Setting dwc_otg.nak_holdoff=0 in cmdline.txt is recommended if you're using devices from this manufacturer.

Still waiting for confirmation that this has changed the behaviour on the other device...

@edwar64896
Copy link
Author

@malacalypse yes the kernel configuration parameters have always gone into /boot/cmdline.txt

@edwar64896
Copy link
Author

I'm pretty sure that the device I am using has an internal hub as the usb gadget function it exposes is switchable in the config. You can either have it running as a serial device or as a mass storage device. Will this affect the way that the RPi sees it?

@malacalypse
Copy link

@P33M Ok, good to know, and thank you for the explanation. With this configuration my setup appears stable, so I'll continue with that!

@edwar64896
Copy link
Author

Update:

Ok so to keep things moving, I built an RS232 to TTL converter, enabled the onboard UART and connected it all up via the onboard UART.

Situation is much worse than with the USB! literally every single packet is losing data.

Does this give anyone any more of an idea as to what is happening? I am completely stumped.

@pelwell
Copy link
Contributor

pelwell commented Oct 18, 2018

That doesn't sound right.

The BCM2835 family of devices have 2 UARTs:

  • UART0 is an ARM PL011 variant, and appears as ttyAMA0.
  • UART1 is an 8250 clone with no flow control and tiny FIFOs. It has the additional restriction that the baud rate clock is derived from the VPU core frequency, which means the core frequency must not be allowed to change. enable_uart=1 does an implicit core_freq=250 on BT-equipped Pis.

UART0 is normally used for Bluetooth on Pis that have it, otherwise it is the console UART. /dev/serial0 is (by default) an alias for the best available UART. You can swap the roles on a BT Pi and free UART0 for other applications with either dtoverlay=pi3-miniuart-bt or dtoverlay=pi3-disable-bt, depending on whether or not you need Bluetooth.

Pi 3's BT didn't wire up the flow control signals to the modem for pin count reasons, and we found that 921600 baud was about the highest we could run UART0 reliably (this was after I fixed a few bugs in the UART driver). For UART1 we settled on 460800 baud. On Pi Zero W and 3B+, which have RTS and CTS connected, you can run UART0 at 3Mbaud with no loss of data. Using an overlay or some GPIO manipulation program (e.g. raspi-gpio) you can map UART0's RTS and CTS signals onto GPIO 16 & GPIO 17: raspi-gpio set 16-17 a3 should do it. Then be sure to enable flow control on the UART - something like stty -F /dev/ttyAMA0 3000000 crtscts raw -echo. Also make sure there is no console enabled on the port - remove console=ttyAMA0,... or console=serial0,... from cmdline.txt.

@malacalypse
Copy link

I spoke a bit too soon. I still have significant drops when I have the MIDI data go in the Keystep, out to another device (like a sequencer), back in from that device, and out again to the ultimate synth endpoint. So it seems like the timing on the USB interface has a strong effect on the reliability of the bulk data transfer here. If I get time I'll drop back a few kernel versions to see if that solves the issue.

@JamesH65
Copy link
Contributor

@P33M Any recent changes that may have helped here?

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

@JamesH65 JamesH65 added Close within 30 days Issue will be closed within 30 days unless requested to stay open Waiting for internal comment Waiting for comment from a member of the Raspberry Pi engineering team labels Jul 31, 2019
@JamesH65
Copy link
Contributor

Closing due to lack of activity. Please request to be reopened if you feel this issue is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Close within 30 days Issue will be closed within 30 days unless requested to stay open Waiting for internal comment Waiting for comment from a member of the Raspberry Pi engineering team
Projects
None yet
Development

No branches or pull requests

6 participants