Degraded network performance after recent update #4133
Comments
|
Could you please update your issue with details for the used UASP USB3 adapter from dmesg? example for detail I'm looking for (note: this device is not affected!) and output of |
|
I've attached the output below. |
|
Update 10Feb2021 16:49 UTC |
|
64MB file? Try again with a larger file and several read/ write operations. |
|
I just tested it again. Flashed the newest Raspberry Pi OS 32bit version (still running the old 5.4.83-v7l+ Kernel ), configured a basic samba share and ran a few tests with around 10-20GB of data. After that, I updated to the kernel version 5.10.11-v7l+ and updated the preinstalled packages as well. This confirms that it's not related to a config issue or some other factor. |
|
Can you test if the issue is samba related or network related? |
|
It doesn't seem to be network related. Edit: Samba itself shouldn't be the issue either, since the version hasn't changed with the update. |
|
I am having a similar issue with my NFS shares that I am using for my kodi box. Everything was working just fine, but at the moment the video playback is impossible. My rpi4 is currently running: Linux 5.10.14-v7l+ #1401 SMP Mon Feb 8 14:27:07 GMT 2021 armv7l Before the update everything was working fine. Strangely enough, when accessing the share from my windows laptop, on both the video playback is smooth as butter. This is strange as for example iperf3 shows stable 900 Mb/s+. RPI4 nfsstat after 2min: RPI3 after 2 min playback: |
|
@dq-git your issue may be xbmc/xbmc#19147 |
|
I have the same problem with my Pi 4 4GB! I use it as a NAS with an 8TB WD Elements attached. |
|
This is also the second time I'm noticing IO errors after the kernel update. |
|
@popcornmix Thanks, it looks like this is just the case here. Meanwhile I reverted kernel to the same that was working on rpi3 and magic happened, everything works now. |
|
Same bug. Upgrated to kernel 5.10.11 speed: 70 MB/s. Pi 4B 2GB + WD Elements 5TB USB 3.0 |
New test: It is not the kernel, it is another update. Continuous speed problem, after updating everything except the Kernel. My OMV, is working with an old kernel, and after updating everything else, the problem starts. |
|
Oh.. I was quite sure it’s related to the kernel. |
|
Confirming exactly which package update causes the reduced performance would be helpful. |
|
I'll try to see which package reduces the performance. Edit: Looks like it's indeed not related to the kernel. @popcornmix Do you have an idea which of those packages might have an impact? |
|
This combo has the bug. In a Fresh Install, if you update this:
When the server is restarted and changes are applied, then the speed is reduced. We have to do the test by omitting the SYS-Mod, which is the only one with a description that mentions the performance of the system. |
|
I don't think it's one of those packages. So it leaves those packages: |
|
I think you should be able to rule out the following by upgrading them all together: |
|
Test 3. Test performed in a completely new environment.
The following additional packages will be installed:
The following packages will be upgraded:
Just by updating those packages, it is already automatically reflected in performance. The update was done with the 5 packages simultaneously, not one by one. |
|
Well, this is super strange. I've updated all packages one by one but the issue didn't occur or at least I didnt notice it. The Pi didn't throttle, whatsoever. @jorensanbar Is your Pi overclocked? Edit: But I'm still seeing performance issues when downloading files from the samba server. So it might indeed be one of those packages @jorensanbar mentioned |
|
@iandk , negative, my Pi is Stock I'm going to reinstall the Pi Os Stock image again, and do the test skipping the SYS-Mod update. I don't know what other option I have. |
|
Are you seeing those issues with both write and read speed? I'll also reinstall the Pi again and test if one of the packages you mentioned might be the problem. |
|
Does anyone of you know if any of the recent updates changed some behavior that could explain this performance drop? |
|
The xbmc issue @popcornmix linked to ended with this patch appearing in the upstream kernel: |
|
P.S. The latest rpi-update kernel is now on 5.10.16, so might be worth a try. |
|
Thank you, how can I update to 5.10.16? With the rpi-update command? At least the read performance issues seem to be somehow related to the kernel, I've done a clean reinstall and tested the read/write performance. Write speed seems to be fine so far (at least without OC) even after updating/ upgrading all packages and the Kernel. Read performance started to drop with the Kernel upgrade. |
|
|
|
Yep, I've just updated to the latest kernel. |
|
There's been no progress on our end - without some insight to guide the investigation we are unlikely to invest any more effort into it. |
In my case, the only solution I have is to use Raspberry Pi OS Debian version: 10 (buster), with Kernel 5.10.103-v7l+. Keep speeds at their maximum via SMB. I can't use Debian 11, because I can't handle the idle state of hard drives. |
|
Just for your information:
As this issue might relate to Debian 11 and the involved updates I tried to find some information in the debian world - with no success. |
|
|
|
A few weeks ago I tested overlocking the PI + using a EXT4 mounted SSD via samba. I was able to write with about 100-110MB/s but the read speed was always much slower and maxed out at 75MB/s this was definitely not the case with 5.4 |
|
I have to report: after updating windows 11 to 22H2 I can write to OMV on a smb share with appr. 100 MB/s -> so it works between my OMV server and my windows 11 machine as before. I have no clue why. |
|
Hello, I also have a Raspberry Pi and I can not exceed 73 MB / s transfer speed accessing from windows to my samba on the raspberry. |
|
I continue the previous comment. I also tried other File Browsers for Windows and was surprised that I got better speeds reaching 90-100 MB/s, when Windows Explorer could not exceed 73 MB/s. |
Hi, Possible so yes but on my setup the file browser seems not a variable. |
|
I report the same drop in performance when I upgraded my Raspberry PI4 2GB. Then I decided it was time to update to the latest release of the OS, i.e. bullseye from After I tried starting off from a clean installation of buster, I added bullseye APT repos and installed samba with Further I tried something interesting. I picked the kernel from bullseye (basically the 1st partition) but the root filesystem from buster (2nd partition). I had to copy the modules from bullseye to match the kernel version into the rootfs. Finally, what remained to try was to test install samba But, in conclusion, I think we can assume with a high degree of confidence that this drop in performance has to with samba rather than something else. Changing the kernel didn't affect the issue, unlike samba, swapping versions has implications in performance. |
|
samba 4.17.3 shows the same behaviour, fast write, slow or unstable read |
|
Having same issue, except I'd kill for 70mb/s, only getting 10-12 mb/s. |
|
are you using ntfs ? |
Are you using USB Devices? Use this command in terminal: lsusb -t This shows you the way the device was connected, all should be at minimum USB 3.0 to take advantage of the speed (5000M)
|
|
Gave it another try, even when overclocking to the max it’s still showing the same issue. |
|
So Ubuntu has the same issue. Wanted to try Fedora 37 but it has a usb
driver bug so that if anything is plugged into the usb2 ports the system
can't boot... back to pi os.
Using current buster, lite version, ran updates, installed zram, smbd,
OC, static IPs on wlan0 (5ghz) and eth0 (directly wired to PC). Shared
drive is usb3 120gb on blue port, btrfs (was using ext4 and exfat
previously), headless with hdmi, bt and usb2 turned off to conserve heat
and power. Tried the legacy usb interrupts with no change.
Ran perf tests, everything sings along perfectly on all connections
using anything but smb.
Got an interesting result on a fresh boot, testing on a mix of files
over eth0:
For a brief moment it worked like it use to, peaking at 115 mb/s.
Then back to shenanigans. Note: this doesn't show how it drops to 0 mb/s
for long periods. the actual transfer time for 10ish gb is about 30 min.
top - 14:59:42 up 19 min, 1 user, load average: 2.19, 5.90, 4.33
Tasks: 132 total, 3 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 8.2 sy, 0.0 ni, 84.0 id, 5.3 wa, 0.0 hi, 1.9 si,
0.0 st
MiB Mem : 3844.3 total, 3509.8 free, 108.4 used, 226.0 buff/cache
MiB Swap: 961.1 total, 961.1 free, 0.0 used. 3657.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
72 root 0 -20 0 0 0 R 24.2 0.0 0:05.67
kworker/u9:0+brcmf_wq/mmc1:0001:1
915 pi 20 0 100484 20072 16304 R 12.9 0.5 0:13.21 smbd
It sure looks like something is up with kworker and smbd. Both jump
around and stay high with smbd going up to 70% CPU when the transfer
drops to 0 mb/s. You can still see the load avg peak. Also id is way
high for a file transfer indicating the cpu is getting pounded for no
reason.
I guess its time to figure out back versions with apt.
Some one asked for lsusb:
***@***.***:~# lsusb
Bus 002 Device 002: ID 0781:5583 SanDisk Corp. Ultra Fit
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Happy to share other config files if any one is curious.
I'm a sys admin so if you have anything to try, feel free to make
suggestions.
I'm probably going to try an strace/ltrace and see if anything jumps out.
…On 12/9/2022 2:20 PM, Ian wrote:
Gave it another try, even when overclocking to the max it’s still
showing the same issue.
I can also confirm it’s not related to the SMB implementation in
windows as macOS shows the exact same problems.
—
Reply to this email directly, view it on GitHub
<#4133 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM76KAW4PZAOQTSTFA3QMOLWMOPARANCNFSM4XJKWXBQ>.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Using:
strace -f -p (pid for smbd) > strace.out 2>&1
is flooded with epoll_pwait messages.
Turned off wifi and kworker is no longer showing up so it is definitely
a smbd issue and it was also impacting the wifi driver.
apt install trace-cmd
trace-cmd record -e workqueue:workqueue_queue_work
trace-cmd report > trace.log
grep -o -e "function=[_a-zA-Z_][_a-zA-Z0-9]*" trace.log|sort|uniq -c
|sort -rn
1036 function=blk_mq_run_work_fn
92 function=vmstat_update
89 function=dbs_work_handler
Hmm, looks like maybe Multi-Queue Block IO Queueing Mechanism (blk-mq) bug.
Turning off multi queueing for scsi (add scsi_mod.use_blk_mq=0 to
cmdline.txt) and wifi is off.
No joy. checking a new strace. Still flooded with:
epoll_pwait(3, [{EPOLLIN, {u32=2641603648, u64=367713823808}}], 1, 213,
NULL, 8) = 1
recvmsg(50, {msg_name=NULL, msg_namelen=0,
msg_iov=[{iov_base="\254\305\260\v\273\234\200\367:\230\371d[S\250\302\2052\302'\360\370\300_\213)#\271\335\303\203Y"...,
iov_len=374584}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 45260
Out of ideas other than going to a back version of smb. :(
…On 12/9/2022 2:20 PM, Ian wrote:
Gave it another try, even when overclocking to the max it’s still
showing the same issue.
I can also confirm it’s not related to the SMB implementation in
windows as macOS shows the exact same problems.
—
Reply to this email directly, view it on GitHub
<#4133 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM76KAW4PZAOQTSTFA3QMOLWMOPARANCNFSM4XJKWXBQ>.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Version 12 Bookworm is out, but no changes in this regard. Occurred to me trying different services for sharing files, like NFS and KSMBD (now available on bookworm, it's a kernel module that implements a SMB server, is replacement of samba and promises better performance). However I got very similar results using the 3 services, around 60MB/s reads. |
|
I tested it with Bookworm on a Pi 5. But read speed varies quite a lot (80-100MB/s). |
|
I am experiencing the same issue as well. OS - Bullseye Drives formatted to ext4 I used to be able to read at around 100+ MB/s, but at some point along the update cycle, this drastically changed. |
|
Good luck getting them to admit it. I ended up shelving the nas.
…On Sunday, February 11, 2024, name01019 ***@***.***> wrote:
I am experiencing the same issue as well.
OS - Bullseye
Hardware - Raspberry Pi 4
Drives formatted to ext4
Read speeds over wired ethernet - 30 MB/s
Write speeds over wired ethernet - 112 MB/s
I used to be able to read at around 100+ MB/s, but at some point along the
update cycle, this drastically changed.
—
Reply to this email directly, view it on GitHub
<#4133 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM76KAVLK6MCR7C5BIGT4ELYTCGB5AVCNFSM4XJKWXB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTG42DSMRSGQ3A>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
There's no lack of admission, it's the understanding that is absent. To repeat my comment from above:
Feel free to point us to the analysis that somebody has done on this open source software clearly demonstrating the cause of the slow down - memory fragmentation, solar storms, etc. - and we'll apply any available fix. |
|
Same here, pi4, 64bit with Linux 6.1.0-rpi8-rpi-v8. I'm not able to hunt it down what causes this issue, but I'm sure I can reproduce it. Like the others the last version which was working fine was 5.4.79-v71+. But... this specific problem does not happen always. For example few hours back I had this problem with around 80MB/s read from pi, even connected my laptop via ethernet to check, but was the same. Then I restarted smb service few times and speeds are now back to normal (105-110MB/s). Noticed it also earlier that if you recycle smb few times, then it sometimes goes back to normal state and stay for a longer time being fine. if you have suggestion on what to check - happy to do that edit: got hit again, this time limited to 60MBs. Restarted smbd - back to normal. |
|
I'm doing a little investigation with Pi 5 as well, because I figure SMB should be able to hit at least a gigabit reliably writing to the Pi if the disks are fast enough... which I've had the same issue with 900 MB/sec NVMe and SATA SSDs now, so I don't think it's storage or the PCIe bus, but something in networking or Samba. I'm doing some testing in this issue: geerlingguy/raspberry-pi-pcie-devices#615 |
I think it's for sure Samba issue, as for me |
|
I've been measuring with large file copies 50-100 GB in size, and for the first few seconds, all copies start off around 115 MB/sec, then they start going up and down between 50-110 MB/sec, averaging about 70 MB/sec with ZFS RAIDZ1 (96 MB/sec on mdadm RAID 0). I tried rebooting the Pi and quickly starting the copy, I tried letting it run for 30 minutes, doing a copy, then restarting smbd and trying the copy again, makes no difference on this Pi at least (Pi 5 8 GB RAM). I'm digging into samba settings too, to see if there are any other tweaks that could help, or performance monitoring tools that could find where the bottleneck is (could even be client side... need to spin up a few containers and run through Linux too, not just macOS). |
|
I got around 100/110 MB/s by modifying smb.conf in the aio read size and aio write size part, setting it to 0. This is my smb.conf: |
|
@falacu - I tried that, restarted everything, and it's still giving me the inconsistent speeds on macOS — apparently it's giving me good, consistent 110 MB/sec speed over in Windows 11, so at this point, I'm trying to see what about Apple's implementation could be breaking things (see the issue I linked earlier, I've been spelunking pretty deep). |
Describe the bug
I'm using a Pi 4 as a basic Samba share for several months now.
Recently after performing a system update & upgrade I noticed a degraded read and write performance.
It must be related to a recent update since otherwise, nothing has changed.
The write performance has dropped from a constant 113MB/s to around 75MB/s.
Since nothing else has changed in my setup I flashed an older build of Raspberry Pi OS ( 64bit was the only image I could find) to the Pi.
With this version, it's performing very well, just like the last few months.
But after running a system update && upgrade performance is significantly worse again.
The Samba version is exactly the same in both situations.
I guess it's related to the newer kernel and maybe some changes to the underlying network stack?
To reproduce
Configure a basic samba share on some external storage with an older version of Raspberry Pi OS.
I've used this build
After running a system update && upgrade the write performance is much lower than before.
Expected behaviour
More or less stable transfer at 100MB/s+
Actual behaviour
After a few seconds, the write speed to the Pi drops to around 75MB/s and stays there.
System
Raspberry Pi 4 2GB with Crucial MX500 attached via UASP USB3 adapter.
The disk is formatted as XFS and has trim support enabled.
WORKING
NOT WORKING
Logs
No relevant output in dmesg or syslog
Additional context
I've tested it with three different drives, and two different Pis.
I also reinstalled Raspberry Pi OS several times, with nothing other than samba installed.
Therefore I'm 100% sure it's related to one of the updates.
The disk itself is performing well in both cases.
The text was updated successfully, but these errors were encountered: