iperf3 show a lot packet loss for UDP test when bandwidth is 500M but no packet loss when bandwidth is 5000M

[ryanlyy]

NOTE: The iperf3 issue tracker is for registering bugs, enhancement
requests, or submissions of code. It is not a means for asking
questions about building or using iperf3. Those are best directed
towards the iperf3 mailing list at iperf-dev@google-groups.com or
question sites such as Stack Overflow
(http://www.stackoverflow.com/). A list of frequently-asked questions
regarding iperf3 can be found at http://software.es.net/iperf/faq.html.

Context

• Version of iperf3:
  [root@test-95-a ~]# iperf3 -v
  iperf 3.1.7

• Hardware:
  [root@test-95-a ~]# ethtool -i eno1
  driver: be2net
  version: 11.4.0.0r
  firmware-version: 12.0.1110.11
  expansion-rom-version:
  bus-info: 0000:37:00.0

• Operating system (and distribution, if any):
  [root@test-95-a ~]# uname -a
  Linux test-95-a 3.10.0-957.el7.x86_64 setting of window size should be explicit #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Please note: iperf3 is supported on Linux, FreeBSD, and macOS.
Support may be provided on a best-effort basis to other UNIX-like
platforms. We cannot provide support for building and/or running
iperf3 on Windows, iOS, or Android.

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.):
  two hosts are in same subnet

[root@test-95-a ~]# ethtool eno1
Settings for eno1:
Supported ports: [ Backplane ]
Supported link modes: 10000baseKR/Full
20000baseKR2/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseKR/Full
20000baseKR2/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000000 (0)

    Link detected: yes

Please fill out one of the "Bug Report" or "Enhancement Request"
sections, as appropriate.

Bug Report

I am testing my host network bandwidth on UDP. when my bandwidth is 500M, there are a lot packet loss (20%). But interesting is: when I increase bandwidth to 5000M, there is no packet loss.
Of course I can fix 500M packet loss by setting device txqueuelen from 1000 to 10000. The issue here is when txqueuelen is 1000, there is packet loss in if bandwidth is 500M but no packet loss if 5000M.
When packet loss in 500M, I found "send buffer error" count of netstat -su is increasing.
Others kernel parameter are same.

• Expected Behavior
  There is no packet loss when bandwidth is 500M.
• Actual Behavior
  There is packet loss when bandwidth is 500M but no packet loss when it is 5000M.
  [ 5] 1.00-2.00 sec 47.1 MBytes 395 Mbits/sec 0.001 ms 14683/69612 (21%)
  [ 5] 2.00-3.00 sec 47.5 MBytes 398 Mbits/sec 0.001 ms 14269/69583 (21%)
  [ 5] 3.00-4.00 sec 48.4 MBytes 406 Mbits/sec 0.001 ms 12931/69270 (19%)
  [ 5] 4.00-5.00 sec 50.0 MBytes 419 Mbits/sec 0.003 ms 11422/69656 (16%)
  [ 5] 5.00-6.00 sec 48.5 MBytes 407 Mbits/sec 0.002 ms 12995/69500 (19%)
  [ 5] 6.00-7.00 sec 48.3 MBytes 405 Mbits/sec 0.001 ms 13103/69372 (19%)

[ 5] 124.00-125.00 sec 253 MBytes 2.12 Gbits/sec 0.004 ms 0/295087 (0%)
[ 5] 125.00-126.00 sec 263 MBytes 2.20 Gbits/sec 0.005 ms 0/306046 (0%)
[ 5] 126.00-127.00 sec 255 MBytes 2.14 Gbits/sec 0.002 ms 0/296643 (0%)
[ 5] 127.00-128.00 sec 253 MBytes 2.12 Gbits/sec 0.002 ms 0/294881 (0%)
[ 5] 128.00-129.00 sec 252 MBytes 2.11 Gbits/sec 0.002 ms 0/293689 (0%)
[ 5] 129.00-130.00 sec 253 MBytes 2.12 Gbits/sec 0.001 ms 0/294714 (0%)
[ 5] 130.00-131.00 sec 252 MBytes 2.12 Gbits/sec 0.002 ms 0/293909 (0%)

• Steps to Reproduce
  SERVER: iperf3 -s -i 1
  CLIENT: iperf3 -u -c 10.67.48.95 -l 900 -b 500M -i 1 -t 0 -P 1
  iperf3 -u -c 10.67.48.95 -l 900 -b 5000M -i 1 -t 0 -P 1
• Possible Solution
  My workaround on this issue is to increase txquenelen from 1000 to 10000.

Please submit patches or code changes as a pull request.

Enhancement Request

• Current behavior

• Desired behavior

• Implementation notes

If submitting a proposed implementation of an enhancement request,
please use the pull request mechanism.

[ryanlyy]

NOTE: this packet loss is caused by sending side instead of receive side.

[bmah888]

Thanks for giving lots of good info. Unfortunately I don’t have any immediate insight on this problem. The only observation I have is that in your 5000M test, you’re getting well less than that amount of user-level throughput according to the output. It looks like that’s on the receiver side since it has packet loss stats. Only 2Gbps instead of 5Gbps. I’m not sure how relevant that is to the problem you are experiencing, if at all.

Definitely a head-scratcher!

[ryanlyy]

@bmah888 Thanks for your response. I did another test. iperf works well (it matches my expection, 500M 0 drop and 5000M drop a lot after tunning some parameter, no drop again although iperf only can send 3.3G around same between sender and reciever). so believe there is something wrong with iperf3.

NOTE, as what I have comment, previous test using iperf3, there is no any error in receive side. sending side UDP drop count(netstat -su) really matches what is reported drop in receive side.

[ryanlyy]

Any idea on this issue?

[bmah888]

At this point, the only comment I would have would be to try a newer version of iperf3 (3.7 is current as of this writing). Versions newer than about 3.2 had some fixes related to UDP tests. The results you are seeing are counter-intuitive, but we haven't seen anything like this. Going to close this issue, it can be re-opened if needed.