usbipd lists not all available devices

[dhutter]

(Win 10 pro, 21H2; usbipd v. 1.30)

First, thanks for this nice piece of code !

I have experienced the problem with "usbipd wsl list" that dependig on the port the usb device is plugged in, the usbipd command lists the device or not. In contrast, the powershell command "Get-PnpDevice -PresentOnly" lists it independent of the port used.

The device is a dvb-2 USB-card (TT s2-4600). Its original Windows driver has not been ported to Win10 but causes an instant bluescreen now. Therefore, I deinstalled the entire driver and the device is no longer operational in Win 10.
Get-PnpDevice returns (regardless of the chosen USB-port) the output "Error dvb-s2 USB\VID_0B48..."
Now, the idea is to use the Linux-driver integrated in Ubuntu to use the device. If I plug the card into the front plug (USB-2), usbipd correctly lists the device:
BUSID DEVICE STATE
3-1 dvb-s2 Not attached
7-3 USB-Eingabegerät Not attached
7-4 Microsoft-Hardware - USB-Maus Not attached
and I can attach it to WSL2, record successfully any TV-broadcasting (using vdr) and store the recording back in the windows filesystem. However, once I use the rear plugs (USB-3) and execute "usbipd wsl list", the device is lost (although Windows knows about it as said before).

Any idea what is going on?

[dorssel]

@dhutter
Your problem is most interesting, and new...
Are you able/capable/willing to run the software from source in a debugger, such as Visual Studio or Visual Studio Code? If so, I can provide a number of instructions for you to test. If not, I would have to make a special branch that provides more debug output for the "list" command.

Essentially, the algorithm is as follows: enumerate all USB hubs, enumerate all their children, exclude the hubs themselves, retrieve a bunch of properties for each device -> list. Any of these stages could be at fault:

• Maybe the USB3 hub is not enumerated at all. If you plug other devices in that hub, are they being listed?
• Maybe the device does not expose an expected property when plugged into that port? But which one?
  Too much that could be at fault. This requires proper debugging...

[dhutter]

Meanwhile, I cloned the Git-repro and installed VC2022, so I'm ready to go.

However, regarding the DVB-card, the recorded stream has a lot of dropouts (especially in HD-TV). Looking at the logs (dmesg), I got dozens of error messages like:

[10106.442542] vhci_hcd: unlink->seqnum 772023
[10106.442547] vhci_hcd: urb->status -104
[10106.443390] vhci_hcd: unlink->seqnum 772026
[10106.443394] vhci_hcd: the urb (seqnum 772026) was already given back
[10106.443636] vhci_hcd: unlink->seqnum 772028
[10106.443639] vhci_hcd: the urb (seqnum 772028) was already given back

Are these messages caused by an overload of usbip? A typical HD-stream produces about 7 GByte per hour. Is it just too much data squeezed through the IP channel?

[dorssel]

@dhutter
These messages are normal vhci debug messages (which for some reason are turned on by default on Linux). They could be an indication of congestion, but they are normal STALL recoveries which happen all the time on USB.

For bulk endpoints I've reached 100+ MiB/s on a USB 3 flash drive, so bandwidth is fine. But timing/latency may be a problem for high rate isochronous endpoints. I never did more than bluetooth audio, which is a much lower rate than HD video.

Is the device using isochronous endpoints? You can check with usbview.exe, part of Windows SDK.

[dhutter]

Is the device using isochronous endpoints?

it does. Bus speed is high but not SuperSpeed. It has several endpoints with isochronous transfer type . The highest has 3 transactions per microframe with about 1KBytes max. I've attached the output of usbview in case you are interested in some details.

s2-4600.txt

[dorssel]

@dhutter
Isochronous endpoints can explain the stutter. Going over IP already introduces some latency. But on Windows there is also the kernel mode/user mode context switch. USBIP on Linux is handled entirely in kernel, but on Windows everything goes to user mode as well. So, the data path is:
USB hub stack (kernel) -> VBox stub driver (kernel) -> usbipd-win (user) -> network stack (kernel) -> VM bus (kernel) -> Linux network stack (kernel) -> Linux USB stack (kernel)

usbipd-win has not (yet) been optimized for performance, especially not for isochronous endpoint which were added last. You could try to increase the process priority via Windows Task Manager (details view, usbipd.exe, right click -> High, or even Real Time).

[dhutter]

I think I located the source of trouble concerning the recognition of the USB device:
Rear and front hubs assign different names to the device. In the case of the rear hub the device name of the DVB-card is "Port_#0005.Hub_#0007" and only the description refers to "DVB-S2" while in the case of the front hub both name and description are equally "DVB-S2" (cf. copies of USBDeview outputs). So maybe, your hub-detection condition will erroneously classify the DVB-S2 as a hub and drop it from the list?

Concerning the latency problems of the isochronous endpoints, increasing the priority of the usbipd process shows no significant improvements. Looking at the recorded MPEG2-TS stream, only about one of 50 TS-packets has been dropped (but that's still enough to pixelize each fame). I'm wondering if these packets have been dropped because they arrived out of their designated time slots (for instance, because they are delayed due to some additional concurrent IP-traffic between the host and the VM). Usually, a sufficiently large buffer on the receiver side buffering the packets from the host and forwarding them in regular time slices downstream may help. Is there any option to increase such buffers?

front_hub.pdf
rear_hub.pdf

[dorssel]

@dhutter
About device recognition: I don't see a problem there. But the fact that the 2 hubs name the device differently is interesting. Do both use the default Microsoft hub driver? What is the output of:

reg query "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{36fc9e60-c465-11cf-8056-444553540000}" /v UpperFilters

I could tweak the send buffer in usbipd-win, but I am already setting TCP_NODELAY and TCP sockets nowadays have pretty good default behavior when it comes to window scaling. Also, I agree that the receive buffer is more important here. And on the Linux side, that is all in the kernel. USBIP on Linux is entirely kernel mode, no user process involved. I wouldn't know how to tweak the buffers there...

[dhutter]

the registry does not contain attributes "UpperFilters" or "LowerFilters"; neither for the given class nor for its subclasses of the individual USB devices. The drivers for the front and rear hubs differ. Both are standard Microsoft drivers: the rear hub usbhub3.sys in version 10.1.19041 1202 (from 8/2021) and the front hub usbhub.sys in version 10.1.19041 1 (from 6/2006).

I've read a very interesting but also rather old (2006) paper from Hirofuchi et al. on "USB/IP - a Peripheral Bus Extension for Device Sharing over IP Network". If I understand it correctly, they discuss (in Sec. 4.1.2) in particular how the URB queuing depth (number of URBs that are encoded in a single IP packet) has to match the likely delays introduced by the IP network. This decision is done by the host. However, I'm not sure how far their protocol of USP/IP matches actual specs.

[dorssel]

@dhutter
Then again, how much delay is a VM-bus? < 0.1ms I would say. I think the thread scheduling timing for the usbipd-win user mode process actually causes the most jitter. And I must admit that I am not entirely sure how actual threads map to C# asynchronous tasks, really. I originally designed it for bulk endpoints...

[dhutter]

Then again, how much delay is a VM-bus? < 0.1ms
not at all. On my old Desktop PC a ping between VM and host requires minimal 0.295 ms, maximal 0.58 and in average 0.359 ms. My newer laptop (Lenovo T450s) is even worse: min: 0.327, avg. 0.51, max 0.82 ms ! So the difference between min and max delivery time is more than 0.25 ms (two entire cycles for USB) in both cases.
Does USBIPD bundle URBs in a single IP?

[dorssel]

@dhutter
Could you please try PR #224, the installer is at https://github.com/dorssel/usbipd-win/actions/runs/1734739747.
This could solve the problem (no guarantees...).

[dhutter]

Congrats!
Device is detected on all USB-hubs (and I'm not required to enter my password each time I attach the device to Ubuntu :-)

From my subjective impression, I would say that also the jitter has decreased although HD streams are only stable for about 50 secs then the picture gets pixelized, recovers after some seconds, and then gets pixelized again. Looks like not all TS-packets make it in time. But anyway, this is a different issue.

[dorssel]

@dhutter
Yes, the jitter should have decreased if you were indeed hit by the race condition that could cause the URB replies to be re-ordered. When you see the artifacts appear, does the syslog say anything about STALL or UNLINK?

[dhutter]

dmesg shows a regular sequence of "unlink seqnum.." and a following "urb already given back" message. The occurring seqnums increase typically by 4 till 8.

[dorssel]

@dhutter
UNLINK is generally requested when the requester either thinks it has been waiting for too long (timeout) or when it stops. So it is quite normal when you stop recording. Question is, do these appear while recording, because that would be an indication that the latency is too long.

For example, when I test with bluetooth audio (i.e. a USB bluetooth dongle via usbipd-win to Linux), I normally get soon after starting:

Jan 26 21:46:07 oakdale-ubuntu pulseaudio[2710]: Cannot set requested source latency of 66.67 ms, adjusting to 68.54 ms
Jan 26 21:46:08 oakdale-ubuntu pulseaudio[2710]: Too many underruns, increasing latency to 205.00 ms

And after that, everything is fine. But I don't know how DVB capture feels about underruns/latency/timeout. Maybe it just cancels the current stream (UNLINK) and then starts again. Pulseaudio at least recognizes the latency/jitter and simply increases its buffer windows, after which everything is fine.

[dhutter]

the UNLINK messages occur regularly during streaming (see attached file) .

In DVB it's a bit more complex as there are always multiple, interleaved MPEG transport streams. Each of these streams is a sequence of packets. A packet has 188 bytes and contains a sequence byte that indicates its relative position in the sequence. The USB-device streams this bunch of streams (corresponding to multiple channels and different types (Audio/Video/Subtext/EPG...)) downstream to some software. The timing (e.g. synchronizing A/V) is only done at the very end inside the renderer.
Inspecting the recorded streams, you can easily recognize dropped packets by checking the sequence bytes of the packets. Approximately, 1 of 50 packets is dropped. Interestingly, it takes about 30 sec until the dropping starts (buffer underflow?)

urb.txt

[dhutter]

just for curiosity:
I examined the TS stream for dropouts more carefully and made some interesting observations:

the dropouts are always one (!) single byte. The first byte is lost after 64 MByte of recording and then the next bytes are lost constantly every 21 MByte later (it fluctuates between 21 and 21.07 MBytes). (However, these absolute numbers refer only to the particular channel I recorded (there is another channel streamed on the same frequency and thus also delivered by the DVB-device)

[dorssel]

That's very interesting (and suspect!) indeed. As we all know, there are only 2 types of bugs: null-pointer-dereferences, index-out-of-bounds, and off-by-one-errors. This smells like the latter...
Just FYI: I am embedding USB Pcap for these kind of diagnostics, see #242. No isochronous capture (yet!)...
If you want, you can also get USB captures on the Linux side, using usbmon and tshark (hint!)

[dhutter]

Tshark is fine but what is the name for the usbip interface? usbmon1 does not exist in WSL.

[dorssel]

Ah. I guess that isn't enabled in the WSL kernel. I use hyper-v, with a full Ubuntu kernel. There I can just modprobe usbmon. If you want it in WSL you will have to compile your own kernel.

[dhutter]

ok, compiling the kernel is not a problem (did it before to include all the dvb-usb modules and required firmware). udev is not available/configured in WSL