mwan3: iperf3 bind (-B) does not work properly when mwan3 is enabled

[jamesmacwhite]

Maintainer: @feckert
Environment: OpenWrt 19.07.3 Linksys WRT3200ACM

Description:

I've noticed that when using iperf3 and mwan3 is enabled the bind option of iperf3 is overridden by the mwan3 routing rules. For example, because I have multiple dual stack WAN interfaces, I may run specific tests like this like:

6in4 IPv6:

iperf3 -6 -B 2001:470:xxxx:xx::x -c bouygues.iperf.fr -R

Wireguard VPN IPv6:

iperf3 -6 -B fc00:bbbb:bbbb:bb01::1:611c -c bouygues.iperf.fr -R

Depending on what policy is defined in the default IPv6 mwan3 config in this case, the test will always use that WAN because of this rule:

config rule 'default_rule_v6'
    option dest_ip '::/0'
    option family 'ipv6'
    option use_policy 'wan_wanb'

This seems to override the iperf3 bind option and the outgoing traffic from going to the specified interface. If I disable mwan3 the bind option works.

This doesn't happen when using something like cURL and traceroute when forcing a specific interface.

Is there anything that can be done to make mwan3 and iperf3 play nicely? Not sure this is a "bug" but makes testing with iperf3 difficult with multiple WANs.

Thanks!

[jamesmacwhite]

I guess because iperf3 uses TCP port 5201 you can setup rules around dest_port combined with the src_ip to force the correct WAN interface the traffic should be using but is there a better way?

[aaronjg]

Yes. That is not surprising, but also fairly annoying. Binding to an ip address does not work as reliably as binding to an interface. However, FreeBSD does not support binding to an interface, so a lot of packages that aim to be compatible across platforms do not implement the SO_BINDTODEVICE to bind to the device.

There is basically a hack right now to avoid this with outgoing echo requests so that things work properly for mwan3track:
https://github.com/openwrt/packages/blob/openwrt-19.07/net/mwan3/files/lib/mwan3/mwan3.sh#L397

It would be great to have a more general solution, and to remove this hack. Perhaps for packets that originate on the device, if they have a a src ipv6 address that differs from what would be expected from the default route, then mwan3 should mark them with $MMX_DEFAULT so that they go through the main routing table. However, I don't think that will do quite what is desired, because if you try to bind to the default ipv6 source, it would still go through the mwan3 rules.

[jamesmacwhite]

@aaronjg Ah good to know. I didn't make the connection it would be due to the lack of SO_BINDTODEVICE.

There is a verbose workaround for this, basically targeting a rule at TCP 5201 (default iperf3 port) with dest_port and then basically combining it with a src_ip value which would be the interface address that is being used with iperf3 with the -B option. This then will make sure traffic traverses the right WAN interface, but in my case that's 8 additional rules to make sure iperf3 tests are accurate for all my configured WANs in mwan3, so it isn't ideal, but works for now.

[aaronjg]

There is a verbose workaround for this, basically targeting a rule at TCP 5201 (default iperf3 port) with dest_port and then basically combining it with a src_ip value which would be the interface address that is being used with iperf3 with the -B option

Could you do it with just one rule to tell mwan3 to use the default routing table for all traffic to TCP 5201? Then it will fall back on the default routing table, which will hopefully respect the choice of source address.

[jamesmacwhite]

@aaronjg Ah I forgot about the 'default' value for rules to fallback to the routing table.

Something like this, could in theory condense the rules without having to define one for each WAN?

config rule 'iperf3_default'
	option dest_port '5201'
	option proto 'tcp'
	option use_policy 'default'

[aaronjg]

Something like this, could in theory condense the rules without having to define one for each WAN?

Yep, give that a try and see if it routes things correctly.

[jamesmacwhite]

@aaronjg Seems to work OK! I can tell because my 6in4 WAN can only achieve around 12 Mbits/sec, where as Wireguard can get to over 100 Mbits/sec, which does seem to be happening with this rule and then using -B, so that at least cuts down the rules to force the traffic. Good workaround, thank you!

[aaronjg]

Glad that worked. FWIW, someone has already written a patch to add SO_BINDTODEVICE to iperf3, but it has not yet been merged.

esnet/iperf#817

[feckert]

@aaronjg Thank you for your work.
I will close the ticket, since there is a solution.

[jamesmacwhite]

@aaronjg Interesting, the PR looks abandoned now though (looking at the original date), so I'm not sure if will be implemented anytime soon, but thanks for your advice regarding the workaround, that should work for my requirements.

[ptpt52]

This is a bug in wireguard, I am trying to fix in the code

https://github.com/x-wrt/x-wrt/blob/master/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.patch

[aaronjg]

Glad that worked. FWIW, someone has already written a patch to add SO_BINDTODEVICE to iperf3, but it has not yet been merged.

esnet/iperf#817

Patch has been merged in to iperf3
esnet/iperf@21581a7

[jamesmacwhite]

Didn't expect that! Looks like the PR was stale. I guess once iperf3 is updated in OpenWrt, you won't need to worry about mwan3 use for iperf3 anyway if it properly implements SO_BINDTODEVICE.