Connection Refused in BWCTL when iperf3 got upgraded from 3.3-1 to 3.5-1

[John-McClane]

I'm sorry in advance if this is considered double posting,
but could you please take a look at
https://github.com/perfsonar/bwctl/issues/43

I wasn't sure if you are monitoring the bwctl repo, so I thought I should ask here to.
Feel free to delete or mark as closed the issue anytime.

Thanks in advance for your help.

[bmah888]

Noting this was fixed in bwctl. Marking as invalid, as it was really a bwctl issue, and closing. Thanks for the headsup.

[John-McClane]

I was concerned about iperf3 mainly because the errors started creeping up after the update of iperf3 (not bwctl).

In addition, I keep getting
iperf3 -t 10 -c 90.147.67.252
iperf3: error - unable to connect to server: Connection refused
on any attempt on ANY testpoint, also just after the update occurred.

The connection refused reply is perhaps because (all?) the endpoints have an old (pre 3.5x) version of iperf3?
Note that this is without using bwctl and, seems to me, irrelevant to bwctl in any way.

So my question really is:
If iperf3 errors out on its own, an updated bwctl is not going to make any difference, is it?
The updated bwctl version will likely fail as well, if iperf3 is not establishing a connection, right?
What is the root cause of the connection refused reply of iperf3?

Looking forward to your insight.
Thanks again.

[bmah888]

Reopening. It would have been good to get these other details in your initial bug report. For the record, there really was a bug in bwctl that needed to be fixed, although it might be independent of the bug you reported.

The update from 3.3 to 3.5 should not have affected any of the connection handling, and those version of iperf3 should interoperate. (In general any of the 3.x series should interoperate with each other.)

What kind of systems (OS and hardware) are you using? Is it possible connections are hitting some sort of host-based firewall (such as iptables or pfsense or ipfw?). Is there a NAT between the client and server?

[John-McClane]

Well, I'm a bit confused, as well.

The hardware and software config is as follows:
Hardware: ESXi Virtual Machine (2-cores, 2 GB RAM, 1G ethernet)
Operating system: Linux 4.14.0-3-amd64 #1 SMP Debian 4.14.17-1 (2018-02-14) x86_64 GNU/Linux
Version information: bwctl 1.6.6-1, iperf3 3.5-1, libiperf0 3.5-1

My testpoint has a firewall (iptables), but even if I drop the firewall, the result is the same.
The other endpoint is in another country, and out of my control,
so I'm just guessing there might be a NAT and a firewall involved.

In any case, I have tested several public endpoints, as published at the Lookup Service Directory of PerfSonar.
http://stats.es.net/ServicesDirectory/
And I'm just guessing, but since these are public, they should be accepting incoming connections from everywhere, so NAT and firewall shouldn't be a problem (or should be properly configured), right?

I have tested several (10 or more) endpoints, either with the DNS Names or with their IPs.
Some of them are:
iperf3 -t 10 -c "perfsonar2.na.infn.it:4823" #(IPv4: 90.147.67.252)
iperf3 -t 10 -c "psonar.cis.gov.pl:4823" #(IPv4: 192.68.51.219)
iperf3 -t 10 -c "perfsonar.mesopsl.fr:4823" #(IPv4: 145.238.157.5)
and the result is always:
iperf3: error - unable to connect to server: Connection refused

The confusing part is that while iperf3 errors out (connection refused), bwctl (for a month or so until last week, and ever since the update to 1.6.6-1 from 3 days ago)
is working as expected, utilizing iperf3.
bwctl -T iperf3 -t 10 -O 4 -c 90.147.67.252

iperf3 gets invoked:
iperf3 -c 90.147.67.252 -B xxx.xxx.xxx.xxx -f m -p 5263 -Z --omit 4 -t 10

and the result is:
bwctl: Using tool: iperf3
bwctl: 12 seconds until test results available

SENDER START
Connecting to host 90.147.67.252, port 5263
[ 16] local xxx.xxx.xxx.xxx port 44639 connected to 90.147.67.252 port 5263
[ ID] Interval Transfer Bitrate Retr Cwnd
........
[ 16] 9.00-10.00 sec 61.2 MBytes 514 Mbits/sec 0 2.74 MBytes
........
[ ID] Interval Transfer Bitrate Retr
[ 16] 0.00-10.00 sec 686 MBytes 576 Mbits/sec 21 sender
[ 16] 0.00-10.04 sec 689 MBytes 576 Mbits/sec receiver
iperf Done.
SENDER END

Even if I try to run the exact same command (as observed from ps), the result is still the same.
iperf3 -c 90.147.67.252 -B xxx.xxx.xxx.xxx -f m -p 5263 -Z --omit 4 -t 10
iperf3: error - unable to connect to server: Connection refused

Could it be that the public endpoints permit iperf3 traffic initiated from bwctl and through the bwctl ports but reject iperf3 traffic not initiated by bwctl on any port (even bwctl ports), ie is it a strict firewall rule?
I have to admit, I didn't consider testing standalone iperf3 connection (without bwctl) before the 3.3->3.5 update, so I have no idea if this is a regression or it was this way all along.

Anyway, hope it helps.
Let me know if there's anything else I could do to help.
Thanks and keep up the great work.