iperf Throughput with ZERO stalls seeing on OTBR-posix latest tag [thread-reference-20230710]

[ravimannem123]

Hello,

Have upgraded the OTBR-posix to the latest tag [thread-reference-20230710].
With RCP design, while setting up the OTBR-posix, Open Thread core tag is being pulled [thread-reference-20230706].

With this(RCP Model), i am seeing the Zero stalls in the throughput whenever i try to run iperf traffic between the Leader and Child.
Please share your thoughts on this ..!

[jwhui]

Zero stalls is good thing, correct? Can you clarify your concern?

[jwhui]

I interpreted "zero stalls" as "no stalls" or "the absence of stalls". But I now understand that you meant it differently.

[ravimannem123]

Hi @jwhui ,
Please find the iperf test summary for reference about the zero stalls.

[root@localhost exec]# iperf3 -s -p 5123 -i 1 -V  -B fd00:db9:0:0:7ad7:67e3:893:b6ec
iperf 3.1.3
Linux localhost.localdomain 4.8.6-300.fc25.x86_64 #1 SMP Tue Nov 1 12:36:38 UTC 2016 x86_64
-----------------------------------------------------------
Server listening on 5123
-----------------------------------------------------------
Time: Thu, 16 Apr 2020 09:44:44 GMT
Accepted connection from fdcf:48d2:c705:1:f231:4e77:f9c1:9a59, port 60152
      Cookie: localhost.localdomain.1721868313.453
[  5] local fd00:db9::7ad7:67e3:893:b6ec port 5123 connected to fdcf:48d2:c705:1:f231:4e77:f9c1:9a59 port 53462
Starting Test: protocol: UDP, 1 streams, 800 byte blocks, omitting 0 seconds, 60 second test
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  5]   1.00-2.00   sec  6.25 KBytes  51.2 Kbits/sec  5364031928.866 ms  0/8 (0%)
[  5]   2.00-3.00   sec  6.25 KBytes  51.2 Kbits/sec  3200822317.555 ms  0/8 (0%)
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec  3200822317.555 ms  0/0 (0%)
[  5]   9.00-10.00  sec  4.69 KBytes  38.4 Kbits/sec  2173147648.372 ms  69/75 (92%)
[  5]  10.00-11.00  sec  4.69 KBytes  38.4 Kbits/sec  1475424173.942 ms  0/6 (0%)
[  5]  11.00-12.00  sec  7.03 KBytes  57.6 Kbits/sec  825388456.601 ms  0/9 (0%)
[  5]  12.00-13.00  sec  4.69 KBytes  38.4 Kbits/sec  560384422.355 ms  0/6 (0%)
[  5]  13.00-14.00  sec  6.25 KBytes  51.2 Kbits/sec  334392314.090 ms  0/8 (0%)
[  5]  14.00-15.00  sec  7.03 KBytes  57.6 Kbits/sec  187067268.557 ms  1/10 (10%)
[  5]  15.00-16.00  sec  6.25 KBytes  51.2 Kbits/sec  111626695.265 ms  1/9 (11%)
[  5]  16.00-17.00  sec  6.25 KBytes  51.2 Kbits/sec  66609833.437 ms  2/10 (20%)
[  5]  17.00-18.00  sec  7.03 KBytes  57.6 Kbits/sec  37263187.847 ms  2/11 (18%)
[  5]  18.00-19.00  sec  5.47 KBytes  44.8 Kbits/sec  23718057.446 ms  1/8 (12%)
[  5]  19.00-20.00  sec  6.25 KBytes  51.2 Kbits/sec  14153040.975 ms  2/10 (20%)
[  5]  20.00-21.00  sec  6.25 KBytes  51.2 Kbits/sec  8445419.620 ms  4/12 (33%)
[  5]  21.00-22.00  sec  7.03 KBytes  57.6 Kbits/sec  4724585.116 ms  1/10 (10%)
[  5]  22.00-23.00  sec  6.25 KBytes  51.2 Kbits/sec  2819265.955 ms  2/10 (20%)
[  5]  23.00-24.00  sec  6.25 KBytes  51.2 Kbits/sec  1682325.455 ms  2/10 (20%)
[  5]  24.00-25.00  sec  6.25 KBytes  51.2 Kbits/sec  1003887.075 ms  1/9 (11%)
[  5]  25.00-26.00  sec  4.69 KBytes  38.4 Kbits/sec  681585.651 ms  2/8 (25%)
[  5]  26.00-27.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  27.00-28.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  28.00-29.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  29.00-30.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  30.00-31.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  31.00-32.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  32.00-33.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  33.00-34.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  34.00-35.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  35.00-36.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  36.00-37.00  sec  0.00 Bytes  0.00 bits/sec  681585.651 ms  0/0 (0%)
[  5]  37.00-38.00  sec  4.69 KBytes  38.4 Kbits/sec  462817.325 ms  129/135 (96%)
[  5]  38.00-39.00  sec  4.69 KBytes  38.4 Kbits/sec  314247.295 ms  0/6 (0%)
[  5]  39.00-40.00  sec  6.25 KBytes  51.2 Kbits/sec  187526.755 ms  0/8 (0%)
[  5]  40.00-41.00  sec  6.25 KBytes  51.2 Kbits/sec  111911.478 ms  0/8 (0%)
[  5]  41.00-42.00  sec  6.25 KBytes  51.2 Kbits/sec  66787.147 ms  0/8 (0%)
[  5]  42.00-43.00  sec  5.47 KBytes  44.8 Kbits/sec  42526.483 ms  0/7 (0%)
[  5]  43.00-44.00  sec  6.25 KBytes  51.2 Kbits/sec  25386.878 ms  0/8 (0%)
[  5]  44.00-45.00  sec  6.25 KBytes  51.2 Kbits/sec  15171.094 ms  4/12 (33%)
[  5]  45.00-46.00  sec  7.03 KBytes  57.6 Kbits/sec  8503.381 ms  2/11 (18%)
[  5]  46.00-47.00  sec  6.25 KBytes  51.2 Kbits/sec  5085.478 ms  1/9 (11%)
[  5]  47.00-48.00  sec  6.25 KBytes  51.2 Kbits/sec  3049.155 ms  2/10 (20%)
[  5]  48.00-49.00  sec  6.25 KBytes  51.2 Kbits/sec  1835.010 ms  2/10 (20%)
[  5]  49.00-50.00  sec  4.69 KBytes  38.4 Kbits/sec  1268.685 ms  2/8 (25%)
[  5]  50.00-51.00  sec  6.25 KBytes  51.2 Kbits/sec  773.748 ms  2/10 (20%)
[  5]  51.00-52.00  sec  1.56 KBytes  12.8 Kbits/sec  690.869 ms  2/4 (50%)
[  5]  52.00-53.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  53.00-54.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  54.00-55.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  55.00-56.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  56.00-57.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  57.00-58.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  58.00-59.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  59.00-60.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  60.00-61.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  61.00-62.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  62.00-63.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  63.00-64.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  64.00-65.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  65.00-66.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  66.00-67.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  67.00-68.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  68.00-69.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  69.00-70.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  70.00-71.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  71.00-72.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  72.00-73.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  73.00-74.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  74.00-75.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  75.00-76.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  76.00-77.00  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
[  5]  77.00-77.52  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  0/0 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-77.52  sec  0.00 Bytes  0.00 bits/sec  690.869 ms  236/491 (48%)
CPU Utilization: local/receiver 0.0% (0.0%u/0.0%s), remote/sender 0.0% (0.0%u/0.0%s)
iperf 3.1.3
Linux localhost.localdomain 4.8.6-300.fc25.x86_64 #1 SMP Tue Nov 1 12:36:38 UTC 2016 x86_64
-----------------------------------------------------------
Server listening on 5123
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
[root@localhost exec]#

[root@localhost exec]# iperf3 -c fd00:db9:0:0:7ad7:67e3:893:b6ec -V -u -p 5123 -i 1 -b 1 -l 800 -t 60
iperf 3.1.3
Linux localhost.localdomain 4.8.6-300.fc25.x86_64 #1 SMP Tue Nov 1 12:36:38 UTC 2016 x86_64
Time: Thu, 25 Jul 2024 00:45:14 GMT
Connecting to host fd00:db9:0:0:7ad7:67e3:893:b6ec, port 5123
      Cookie: localhost.localdomain.1721868313.453
[  4] local fdcf:48d2:c705:1:f231:4e77:f9c1:9a59 port 53462 connected to fd00:db9::7ad7:67e3:893:b6ec port 5123
Starting Test: protocol: UDP, 1 streams, 800 byte blocks, omitting 0 seconds, 60 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   1.00-2.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   2.00-3.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   3.00-4.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   4.00-5.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   5.00-6.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   6.00-7.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   7.00-8.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   8.00-9.00   sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]   9.00-10.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  10.00-11.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  11.00-12.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  12.00-13.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  13.00-14.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  14.00-15.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  15.00-16.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  16.00-17.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  17.00-18.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  18.00-19.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  19.00-20.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  20.00-21.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  21.00-22.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  22.00-23.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  23.00-24.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  24.00-25.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  25.00-26.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  26.00-27.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  27.00-28.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  28.00-29.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  29.00-30.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  30.00-31.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  31.00-32.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  32.00-33.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  33.00-34.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  34.00-35.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  35.00-36.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  36.00-37.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  37.00-38.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  38.00-39.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  39.00-40.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  40.00-41.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  41.00-42.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  42.00-43.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  43.00-44.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  44.00-45.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  45.00-46.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  46.00-47.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  47.00-48.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  48.00-49.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  49.00-50.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  50.00-51.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  51.00-52.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  52.00-53.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  53.00-54.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  54.00-55.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  55.00-56.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  56.00-57.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  57.00-58.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  58.00-59.00  sec  7.81 KBytes  64.0 Kbits/sec  10
[  4]  59.00-60.00  sec  7.81 KBytes  64.0 Kbits/sec  10
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-60.00  sec   469 KBytes  64.0 Kbits/sec  690.869 ms  236/491 (48%)
[  4] Sent 491 datagrams
CPU Utilization: local/sender 0.9% (0.1%u/0.7%s), remote/receiver 0.0% (0.0%u/0.0%s)

iperf Done.
[root@localhost exec]#

[jwhui]

Can you provide the packet sniffer trace and OpenThread logs?

[ravimannem123]

Hi @jwhui,

Please find the packet sniffer trace OT logs for reference..!

OT_tag_20230710_latest_iperf_UDP_traffic_zero_stalls.zip

[jwhui]

From the packet trace, there are a lot of TCP retransmissions and missing IEEE 802.15.4 ACKs. It appears that none of the data sent to 0x1402 has matching IEEE 802.15.4 ACKs. This appears to be an issue at the radio platform layer. Can you indicate which IEEE 802.15.4 radio you are using?