Bad performance - how to tune glorytun?

[frleong]

Hello,

I have an internal network at home, with two routers connected via Mesh Wifi and Powerline Ethernet.

The Mesh wifi uses the subnet 192.168.3.0/24 and the Powerline Ethernet uses the subnet 192.168.11.0/24.

Both routers run on OpenWRT 19.07. One router (Router A) uses the IP address 192.168.3.200 and 192.168.11.200 while the other router (Router B) uses 192.168.11.233 and 192.168.3.233. I have setup Linux policy routing.

I have setup glorytun server at Router B (192.168.11.233) and client at Router A (192.168.11.200/192.168.3.200).

Attached is the configuration, shown from the Router A.

#glorytun show
client tun0:
  pid:    24636
  bind:   0.0.0.0 port 5000
  peer:   192.168.11.233 port 5000
  mtu:    1450
  cipher: aegis256

# glorytun path
path UP
  status:  OK
  bind:    192.168.3.200 port 5000
  public:  192.168.3.200 port 5000
  peer:    192.168.11.233 port 5000
  mtu:     1472 bytes
  rtt:     4.137 ms
  rttvar:  1.621 ms
  rate:    fixed
  losslim: 100
  beat:    100 ms
  tx:
    rate:  25000000 bytes/sec
    loss:  0 percent
    total: 314678 packets
  rx:
    rate:  25000000 bytes/sec
    loss:  1 percent
    total: 91716 packets
path UP
  status:  OK
  bind:    192.168.11.200 port 5000
  public:  192.168.11.200 port 5000
  peer:    192.168.11.233 port 5000
  mtu:     1472 bytes
  rtt:     4.125 ms
  rttvar:  2.091 ms
  rate:    fixed
  losslim: 100
  beat:    100 ms
  tx:
    rate:  12500000 bytes/sec
    loss:  0 percent
    total: 333634 packets
  rx:
    rate:  12500000 bytes/sec
    loss:  2 percent
    total: 91784 packets

# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.208.0.2  P-t-P:10.208.0.1  Mask:255.255.255.255
	          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1450  Metric:1
          RX packets:312648 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1086455 errors:0 dropped:123548 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:376543901 (359.0 MiB)  TX bytes:1429229699 (1.3 GiB)

The link speed of Wifi is about 150Mbps-200Mbps and the Powerline Ethernet about 50Mbps-70Mbps.

#iperf3 -R -B 192.168.3.200 -c 192.168.11.233
Connecting to host 192.168.11.233, port 5201
[  5] local 192.168.3.200 port 45173 connected to 192.168.11.233 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec    0    984 KBytes
[  5]   1.00-2.00   sec  18.8 MBytes   157 Mbits/sec   75   1.19 MBytes
[  5]   2.00-3.00   sec  18.8 MBytes   157 Mbits/sec    0   1.33 MBytes
[  5]   3.00-4.00   sec  21.2 MBytes   178 Mbits/sec    0   1.44 MBytes
[  5]   4.00-5.00   sec  20.0 MBytes   168 Mbits/sec    0   1.53 MBytes
[  5]   5.00-6.00   sec  22.5 MBytes   189 Mbits/sec    0   1.59 MBytes
[  5]   6.00-7.00   sec  22.5 MBytes   189 Mbits/sec    2   1.18 MBytes
[  5]   7.00-8.00   sec  22.5 MBytes   189 Mbits/sec    0   1.25 MBytes
[  5]   8.00-9.00   sec  22.5 MBytes   189 Mbits/sec    0   1.30 MBytes
[  5]   9.00-10.00  sec  22.5 MBytes   189 Mbits/sec    0   1.33 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   213 MBytes   178 Mbits/sec   77             sender
[  5]   0.00-10.07  sec   211 MBytes   175 Mbits/sec                  receiver


#iperf3 -B 192.168.11.200 -c 192.168.11.233
Connecting to host 192.168.11.233, port 5201
[  5] local 192.168.11.200 port 51207 connected to 192.168.11.233 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.11 MBytes  59.6 Mbits/sec    0    324 KBytes
[  5]   1.00-2.00   sec  6.34 MBytes  53.2 Mbits/sec    0    590 KBytes
[  5]   2.00-3.00   sec  6.24 MBytes  52.4 Mbits/sec    0    887 KBytes
[  5]   3.00-4.00   sec  6.25 MBytes  52.4 Mbits/sec    0   1.20 MBytes
[  5]   4.00-5.00   sec  6.25 MBytes  52.4 Mbits/sec   53   1.05 MBytes
[  5]   5.00-6.00   sec  8.75 MBytes  73.4 Mbits/sec    0   1.14 MBytes
[  5]   6.00-7.00   sec  7.49 MBytes  62.8 Mbits/sec    0   1.25 MBytes
[  5]   7.00-8.00   sec  7.50 MBytes  62.9 Mbits/sec    0   1.34 MBytes
[  5]   8.00-9.00   sec  7.49 MBytes  62.8 Mbits/sec    0   1.40 MBytes
[  5]   9.00-10.00  sec  6.25 MBytes  52.4 Mbits/sec    5   1024 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  69.7 MBytes  58.4 Mbits/sec   58             sender
[  5]   0.00-10.17  sec  68.4 MBytes  56.4 Mbits/sec                  receiver

I can successfully ping each other inside the tunnel, e.g. ping 10.208.0.1 from Router A works fine. However, iperf3 via the tunnel has abysmal performance with many retries:

# iperf3 -c 10.208.0.1
Connecting to host 10.208.0.1, port 5201
[  5] local 10.208.0.2 port 47038 connected to 10.208.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.01 MBytes  8.43 Mbits/sec  146   2.73 KBytes
[  5]   1.00-2.00   sec   815 KBytes  6.68 Mbits/sec  135   5.46 KBytes
[  5]   2.00-3.00   sec   900 KBytes  7.37 Mbits/sec  122   10.9 KBytes
[  5]   3.00-4.00   sec   681 KBytes  5.58 Mbits/sec  108   5.46 KBytes
[  5]   4.00-5.00   sec   672 KBytes  5.50 Mbits/sec   94   4.10 KBytes
[  5]   5.00-6.00   sec   744 KBytes  6.09 Mbits/sec  123   4.10 KBytes
[  5]   6.00-7.00   sec   595 KBytes  4.88 Mbits/sec   82   2.73 KBytes
[  5]   7.00-8.00   sec   684 KBytes  5.60 Mbits/sec  120   2.73 KBytes
[  5]   8.00-9.00   sec   677 KBytes  5.55 Mbits/sec  103   1.37 KBytes
[  5]   9.00-10.00  sec   665 KBytes  5.45 Mbits/sec   87   4.10 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.29 MBytes  6.11 Mbits/sec  1120             sender
[  5]   0.00-10.04  sec  7.22 MBytes  6.04 Mbits/sec                  receiver

May I know what I have done wrong? How may I troubleshoot this issue? Thanks!

[TalalMash]

You need to set up tx and rx speed limits in Glorytun per path to fix this issue. Around -10% of max speed is usually good.

[frleong]

You need to set up tx and rx speed limits in Glorytun per path to fix this issue. Around -10% of max speed is usually good.

Hi, tried that already, still bad performance. Any other hints? Do I really need MPTCP to work?

[TalalMash]

You need to set up tx and rx speed limits in Glorytun per path to fix this issue. Around -10% of max speed is usually good.

Hi, tried that already, still bad performance. Any other hints? Do I really need MPTCP to work?

You may need to add artificial latency using TC with different link type on very low latency paths, there is no reordering as well, does using parallel TCP streams increase bandwidth? (E.g iperf -P4)