Pi becomes sluggish/unresponsive using wired Ethernet port with Bravia TV

[mcayland]

I'm using a Raspberry Pi as a NAT router between a wired/wireless networks, and I'm experiencing some serious performance problems with the Model B rev 2.0 on-board Ethernet. This is running the latest version of Raspbian downloaded direct from the main downloads page.

The setup is to allow a remote internet-enabled Sony TV without wireless adapter support to stream internet video from a local wireless router. The wireless adapter is an EdiMax WiFi dongle which works really well on its own; the problem is that when the TV is streaming video through the wired interface, the Pi becomes very unresponsive with ping times on the wireless side rising from around 15ms to 500-600ms. This causes the video stream to freeze every 10-20s when trying to watch streaming video.

Things I've checked:

1. Power problems - I've confirmed with a multimeter that the Pi is getting enough power when both wired and wireless network adapters are running; i've even temporarily replaced FS3 with a wire just to be sure. The power adapter is capable of supplying 1500mA and maintains good voltage, even when the Pi is apparently under load.

2. Latest updates - I've run rpi-update with BRANCH=fiq_split and that doesn't seem to make any difference.

3. Kernel options - I've been through all of the steps except increasing vm.min_free_kbytes listed at http://elinux.org/R-Pi_Troubleshooting#Networking and they haven't made any difference at all.

From reading around the forums, my general feeling is that there are still several issues with the USB drivers which are likely to be related to this problem - can anyone confirm the current status of the USB drivers?

Note I should just add that "top" shows no processes hogging the CPU, which would indicate to me that the excess time is being spent in kernel space rather than user space.

[mcayland]

I did another test whereby I connected my laptop to the Pi instead of the TV, and did a wget on a large file. This showed a sustained transfer speed of around 600K/s and the Pi remained responsive to pings on the wired port.

My thoughts are:

1. Does the wired Ethernet take priority over the wireless ethernet? This may explain why it was the wireless interface that was showing the higher ping times. I'd need another laptop to test the wireless interface though.

2. Could the Bravia IP stack be doing something that causes excessive traffic on the Ethernet port? I don't see how 600K/s should bring a Pi to its knees :/

[popcornmix]

fiq_split branch is now deprecated. The fiq_split code (and additional improvements) are now on the master branch.
Any complaints in dmesg log?

[mcayland]

Thanks for the reply. I didn't realise fiq_split is deprecated - it might be worth someone adding a sticky page for USB to the Pi forums as there is so much out-of-date information related to USB problems and fiq_split on there.

dmesg doesn't show any related error messages at all. Another quick test shows that if I disconnect the cable from the wired interface then the ping times on the wireless interface immediately drop back to normal. I can post the complete dmesg output if required.

Also what speed does the Pi wired ethernet run at? mii-tool on Raspbian reports that it's running at 1000baseTx whereas other posts suggest that the port is only a 10/100baseTx?

[popcornmix]

The ethernet is 10/100. You should see a line in dmesg showing the speed. e.g.
[ 4.921464] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0x45E1

[mcayland]

Ah I see - I think my mistake is that the Broadcom drivers supports the new ethtool rather than the older mii-tool ioctl interface. I'll head over to the Pi and double-check in a bit.

Based on this I have another couple of questions:

1. Is 600K/s USB transfer enough to make a Pi completely unresponsive?
2. Is there a way to measure the impact/performance of USB transfers?

I've seen some posts mentioning that it is possible to count interrupts/sec from /proc, but I'm wondering if there are any further counters specific to the USB/ethernet hub, e.g. no. of retries that would indicate that the Pi is being overloaded.

FWIW if there are monitoring patches like this available, I could probably cross-compile a custom kernel and report back the values if that helps?

[popcornmix]

1. No. The pi can play BluRay level videos at ~50Mbit/s (> 6MB/s) through ethernet or USB mass storage.
2. when transferring data what does top report as idle %

[P33M]

I can replicate this. The problem is likely due to memory allocation thrash. In the extreme case we see stuff like #153 where we end up with actual failures, but in the cases that normally provoke this I assume that the kernel allocator has to do a lot of work to find a contiguous free block of the size requested.

I have yet to profile this and understand it fully - not strictly related to USB, but more likely to happen in conjunction with a USB device plus "bad" driver because most of the bytes in/out of a Pi are usually going to be through networks.

One thing to try: Can you repeat your test with a kernel compiled with the memory allocator set to SLUB?

[mcayland]

@popcornmix : i've just installed vnstat on the pi itself, and while it is sluggish, it seems to be reporting ~1.1Mbps data rate on eth0 and wlan0 when streaming video, so that should be well within the parameters above.

Before streaming starts, top reports idle as being around 96% which drops to around 90-92% when trying to stream video.

[mcayland]

@P33M : glad that you can reproduce. as mentioned above, the incoming data rate from eth0 is about 1.1Mbps. Since most people report that they can use a Pi with no problems as an WiFi hotspot, I wonder if it is something in the smsc95xx driver which produces too many USB interrupts?

From the link you gave above, I tried increasing vm.min_free_kbytes to around 128Kb but that actually seemed to make the ping response times worse during streaming - the times went from around 400-500ms to 700-800ms.

In terms of a kernel to try, I've installed an ARM cross-compiler so if you can point me towards a github repository/branch to try (along with a suitable kernel .config), I'll have a go at trying the different allocator.

Another interesting thing: I've just tried switching the Bravia down to "Normal" resolution (down from its default of high), and the data rate stays at ~1.1Mbps - BUT now I get about 10s of 400-500ms ping times followed by about 10s of normal (~20ms) ping times which seems to be enough to keep the TV from buffering constantly. However the Pi still remains unresponsive during the periods of slow ping response. Does that information help at all?

[P33M]

Use this repo (rpi-3.6.y branch). Config is in arch/arm/configs/bcmrpi_defconfig.

Copy that to .config, then make ARCH=arm menuconfig and under General Setup, scroll down to Choose Slab Allocator -> SLUB.

Build in the normal way - plenty of guides floating around - copy both the modules and the kernel across.

[Ferroin]

@mcayland as has been stated in similar threads, vm.min_free_kbytes is a hint to the virtual memory system to try to keep that much memory free if at all possible; in effect, it's the limit below which the kernel starts swapping pages out to disk to try to free up more memory without a pending allocation. Increasing it above about 16348 (16MiB, because the value is in kiB) will actually decrease preformance for almost all workloads on the Pi because the system will usually spend more time swapping to disk than is really necessary, which is probably why you were seeing such bad latency with it at 131072 (which is what i assume you mean by 128Kb, since it can't be less than 4096 on the Pi).

[mcayland]

@P33M okay i'm just setting up the build environment now. I see there have been some recent USB commits, so should I try a standard kernel (without SLUB) first to use as a baseline?

[mcayland]

@Ferroin thanks for the clarification. I saw the hint about raising vm.min_free_kbytes in one of the Pi Forums which is why I tried it, but as you correctly point out increasing it is not going to help here. There is a lot of out of date/conflicting information around on the forums :/

[mcayland]

@P33M one other thought: has anyone got oprofile to work on the Pi yet? if so, I could try and grab a kernel profile when the Pi is being unresponsive.