iperf3 reporting 0 bytes of throughput under UDP and high latency conditions

[smileyc8]

Context

• Version of iperf3: iperf 3.10.1 (cJSON 1.7.13) Server & Client

• Hardware: Raspberry Pi Model B revision 2 Server & Client

• Operating system (and distribution, if any): 2016-02-03-raspbian-jessie

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.):
  Iperf3 is being used to test across a network of multiple bridges for varying hops from 1-5 to reach the server from the client. The bridges are doing encryption/decryption which adds to the latency and communicating over a wired RF connection. I am using Iperf3 to calculate how the throughput varies over differing hop counts, packet sizes and RF channels. The issue looks similar to iperf3 server udp mode showing 0.0 for everthing #905.

Bug Report

• Expected Behavior
  Seeing current bitrate in all 1 second reports
• Actual Behavior
  Depending on latency, I am seeing 0.00 Kbits/sec, 0.00 Bytes transfered for 1s reports until seeing some data.
  Server
  sudo iperf3 -s -p 2303 -f k

---

Server listening on 2303 (test #1)

Accepted connection from 192.168.105.2, port 52834
[ 5] local 192.168.100.2 port 2303 connected to 192.168.105.2 port 33650
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 10.00-11.00 sec 192 Bytes 1.54 Kbits/sec 1.189 ms 22/25 (88%)
[ 5] 11.00-12.00 sec 0.00 Bytes 0.00 Kbits/sec 1.189 ms 0/0 (0%)
[ 5] 12.00-13.00 sec 3.06 KBytes 25.1 Kbits/sec 35.873 ms 21/70 (30%)
[ 5] 13.00-14.00 sec 3.38 KBytes 27.6 Kbits/sec 10.917 ms 1614/1668 (97%)
[ 5] 14.00-14.65 sec 1.88 KBytes 23.6 Kbits/sec 433.703 ms 16337/16367 (1e+02%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-14.65 sec 8.50 KBytes 4.75 Kbits/sec 433.703 ms 17994/18130 (99%) receiver

Client
ping 192.168.100.2

PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=62 time=178 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=62 time=180 ms
64 bytes from 192.168.100.2: icmp_seq=3 ttl=62 time=179 ms
64 bytes from 192.168.100.2: icmp_seq=4 ttl=62 time=185 ms
64 bytes from 192.168.100.2: icmp_seq=5 ttl=62 time=179 ms
64 bytes from 192.168.100.2: icmp_seq=6 ttl=62 time=185 ms
^C
--- 192.168.100.2 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 178.648/181.499/185.372/2.815 ms

sudo /usr/local/bin/iperf3 -c 192.168.100.2 -t 10 -p 2303 -u -l 64 -f 'k'

Connecting to host 192.168.100.2, port 2303
[ 5] local 192.168.105.2 port 33650 connected to 192.168.100.2 port 2303
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 128 KBytes 1048 Kbits/sec 2047
[ 5] 1.00-2.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 2.00-3.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 3.00-4.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 4.00-5.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 5.00-6.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 6.00-7.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 7.00-8.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 8.00-9.00 sec 128 KBytes 1049 Kbits/sec 2048
[ 5] 9.00-10.00 sec 128 KBytes 1049 Kbits/sec 2048

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1049 Kbits/sec 0.000 ms 0/20479 (0%) sender
[ 5] 0.00-14.65 sec 8.50 KBytes 4.75 Kbits/sec 433.703 ms 17994/18130 (99%) receiver

iperf Done.

and other times it fails to show anything at all
Server

Accepted connection from 192.168.105.2, port 52836
[ 5] local 192.168.100.2 port 2303 connected to 192.168.105.2 port 45091
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 10.00-11.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 11.00-12.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 12.00-13.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 13.00-14.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 14.00-15.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 15.00-16.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 16.00-17.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 17.00-18.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 18.00-19.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 19.00-20.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 20.00-21.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 21.00-22.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 22.00-23.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 23.00-24.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 24.00-25.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 25.00-26.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 26.00-27.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 27.00-28.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 28.00-29.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
[ 5] 29.00-30.00 sec 0.00 Bytes 0.00 Kbits/sec 0.000 ms 0/0 (0%)
iperf3: error - unable to receive control message: Connection reset by peer

Client
sudo /usr/local/bin/iperf3 -c 192.168.100.2 -t 10 -p 2303 -u -l 64 -f 'k'

Connecting to host 192.168.100.2, port 2303
iperf3: error - unable to read from stream socket: Resource temporarily unavailable

• Steps to Reproduce
  Run Iperf3 Server and client with UDP. Set connection between server and client with high latency. As the number of hops/latency increases, The more likely I am to see the "error - unable to read from stream socket: Resource temporarily unavailable". Issue occurs the same if -t is increased from default.

• Possible Solution

[davidBar-On]

I suggest to try some variations of the test to see what happens. That may give more information about the problem:

1. Limit the message rate to 1 or 2 per second, i.e. set -b 512 or -b 1014. If the problem is because of network buffers overflow, this test will be successful.
2. Try using TCP, again with 1 or 2 messages per second. Since the control messages are sent over TCP it may help to understand the problem.
3. If either or both of the above tests are successful, increases the message rate to see in which rate the problem occurs. If there is such case that some of the messages are received and some are not, Wireshark logs may help.

[bmah888]

Also, consider checking for host-based or network firewalls that might be blocking UDP traffic.

[smileyc8]

1. Setting -b 512 and -b 1014 did not seem to make a difference in receiving the control message. Failures still resulted in failures with the bit rate limited.
2. Using TCP with the bit rate limited worked fine. I was able to see the throughput at the limited rate. I am testing UDP specifically otherwise I would use TCP.

The connections between client and server is over a direct ethernet to ethernet connection to the first bridge and rf connection from bridge to bridge. There should be no network firewalls blocking UDP traffic. I don't think this is the issue because UDP traffic does get through for some cases like when buffer length is 64kb.

[davidBar-On]

Setting -b 512 and -b 1014 did not seem to make a difference in receiving the control message.

Is the problem in these cases only with the control messages? If this is the case, what it the throughput on the server side? Can you share the client and server output for one of these cases?

Using TCP with the bit rate limited worked fine.

It may be that for some reason, e.g. because of network loading by other traffic, there are "windows of time" when packets can pass between the client and server. Since UDP does not retry sending, there is a lower chance that it will send when such "window" is open. If you still want to use UDP tcpdump/Wireshark logs may be needed to further evaluate the problem (for both UDP and TCP on the client side).