Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Network performance and bandwidth degradation on Fedora CoreOS #542

Closed
wkruse opened this issue Jun 19, 2020 · 13 comments
Closed

Network performance and bandwidth degradation on Fedora CoreOS #542

wkruse opened this issue Jun 19, 2020 · 13 comments

Comments

@wkruse
Copy link

wkruse commented Jun 19, 2020

We moved half of the machines in our CoreOS cluster to Fedora CoreOS and are running some last and performance tests. Currently we see a severe performance degradation.

We already enabled SMT #181.

Network performance and bandwidth is less then the half of what we measure in our old CoreOS cluster.

$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200601.3.0 (2020-06-16T08:52:21Z)
                    Commit: b51037798e93e5aae5123633fb596c80ddf30302b5110b0581900dbc5b2f0d24
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200601.3.0 (2020-06-16T08:52:21Z)
                    Commit: b51037798e93e5aae5123633fb596c80ddf30302b5110b0581900dbc5b2f0d24
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

All machines have 4 NICs, 2 of which are connected, 10 Gb each.

$ dmesg | grep -E 'eth[0-3]'
[   16.405508] igb 0000:06:00.0: added PHC on eth0
[   16.448153] igb 0000:06:00.0: eth0: (PCIe:5.0Gb/s:Width x2) ec:f4:bb:dd:73:ac
[   16.448459] igb 0000:06:00.0: eth0: PBA No: G61346-000
[   16.631482] igb 0000:06:00.1: added PHC on eth1
[   16.662047] igb 0000:06:00.1: eth1: (PCIe:5.0Gb/s:Width x2) ec:f4:bb:dd:73:ad
[   16.681562] igb 0000:06:00.1: eth1: PBA No: G61346-000
[   43.120673] ixgbe 0000:01:00.0: registered PHC device on eth2
[   43.331544] ixgbe 0000:01:00.0 eth2: detected SFP+: 4
[   43.410192] ixgbe 0000:01:00.1: registered PHC device on eth3
[   43.609709] ixgbe 0000:01:00.1 eth3: detected SFP+: 3
[   44.082559] ixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   44.111132] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   44.186531] ixgbe 0000:01:00.0 eth2: NIC Link is Down
[   44.394567] ixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   44.690565] ixgbe 0000:01:00.1 eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   44.718952] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
[  103.910043] eth0: renamed from vetha5ccb1b
[  223.422011] vetha5ccb1b: renamed from eth0
[  335.393352] eth0: renamed from veth2c3e0d4
[ 3099.597437] veth2c3e0d4: renamed from eth0

Running iperf3 on two machines in Docker on the Fedora CoreOS cluster

{code}
Connecting to host xx.xx.xx.xx, port 5201
[  4] local 172.17.0.2 port 50826 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   574 MBytes  4.81 Gbits/sec  873    624 KBytes
[  4]   1.00-2.00   sec   482 MBytes  4.05 Gbits/sec  832    598 KBytes
[  4]   2.00-3.00   sec   480 MBytes  4.03 Gbits/sec  963    550 KBytes
[  4]   3.00-4.00   sec   472 MBytes  3.96 Gbits/sec  1101    537 KBytes
[  4]   4.00-5.00   sec   501 MBytes  4.20 Gbits/sec  906    602 KBytes
[  4]   5.00-6.00   sec   458 MBytes  3.84 Gbits/sec  768    549 KBytes
[  4]   6.00-7.00   sec   488 MBytes  4.09 Gbits/sec  115    997 KBytes
[  4]   7.00-8.00   sec   508 MBytes  4.26 Gbits/sec  953    376 KBytes
[  4]   8.00-9.00   sec   494 MBytes  4.14 Gbits/sec  434    588 KBytes
[  4]   9.00-10.00  sec   510 MBytes  4.28 Gbits/sec  1301    512 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  4.85 GBytes  4.17 Gbits/sec  8246             sender
[  4]   0.00-10.00  sec  4.85 GBytes  4.16 Gbits/sec                  receiver

iperf Done.
{code}

Running iperf3 on two machines in Docker on the CoreOS cluster

{code}
Connecting to host xx.xx.xx.xx, port 5201
[  4] local 172.17.0.2 port 56838 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.07 GBytes  9.23 Gbits/sec  104    686 KBytes
[  4]   1.00-2.00   sec  1.07 GBytes  9.20 Gbits/sec   32    683 KBytes
[  4]   2.00-3.00   sec  1.07 GBytes  9.19 Gbits/sec   43    609 KBytes
[  4]   3.00-4.00   sec  1.07 GBytes  9.23 Gbits/sec   38    769 KBytes
[  4]   4.00-5.00   sec  1.08 GBytes  9.25 Gbits/sec   23    691 KBytes
[  4]   5.00-6.00   sec  1.07 GBytes  9.23 Gbits/sec   25    659 KBytes
[  4]   6.00-7.00   sec  1.07 GBytes  9.23 Gbits/sec   45    798 KBytes
[  4]   7.00-8.00   sec  1.08 GBytes  9.24 Gbits/sec   16    649 KBytes
[  4]   8.00-9.00   sec  1.08 GBytes  9.24 Gbits/sec   36    725 KBytes
[  4]   9.00-10.00  sec  1.07 GBytes  9.23 Gbits/sec   56    857 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.7 GBytes  9.23 Gbits/sec  418             sender
[  4]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec                  receiver

iperf Done.
{code}

Also after every reboot we see, when we run ip addr show that the devices (eth[0-3]), that are UP change, leading to port that are DOWN when teaming is setup.

Do you have any hints or things we could check to get the bandwith back?
@jlebon
Copy link
Member

jlebon commented Jun 19, 2020

Are you in a position to run the same test using the traditional Fedora 32 Server image? This will at least tell us whether this is specific to FCOS or rather part of the Fedora stack in general.

For metal: https://getfedora.org/en/server/download/
For QEMU/cloud: https://alt.fedoraproject.org/cloud/

@lucab
Copy link
Contributor

lucab commented Jun 24, 2020

Also after every reboot we see, when we run ip addr show that the devices (eth[0-3]), that are UP change, leading to port that are DOWN when teaming is setup.

This is a buggy behavior due to #484, which we are currently working on.

@lucab
Copy link
Contributor

lucab commented Jun 24, 2020 •
edited
Loading

@wkruse can you maybe repeat the same test outside of docker (to pin-point whether it's a container network issue) and with a single link up (to pin-point whether it's a bonding issue, assuming you can test this in your network)?
Also, other than the naming issue above, do you see any suspicious difference in ip link show <IFACE> output across the two OS?

@cgwalters
Copy link
Member

same test outside of docker

It should be sufficient to just --net=host if they aren't already...to the best of my knowledge that takes basically all of the overhead out.

@cgwalters
Copy link
Member

Are you in a position to run the same test using the traditional Fedora 32 Server image?

It seems extremely unlikely to me that something here is specific to the "CoreOS" part of FCOS. Much more likely to be a kernel issue in which case trying different kernel versions via rpm-ostree override replace is what I'd suggest.

@wkruse
Copy link
Author

wkruse commented Jun 24, 2020

@jlebon We are provisioning bare metal machines, using PXE boot, matchbox and ignition. I guess we could PXE boot Fedora 32 Server and install manually to disk. But it looks like, we will not need it for now.

@lucab Thanks for the reference, subscribed.

@lucab, @cgwalters Running the iperf3 containers with --net=host in both clusters, we are getting similar results

Connecting to host xx.xx.xx.xx, port 5201
[  4] local 10.10.224.22 port 54166 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.08 GBytes  9.29 Gbits/sec  211    785 KBytes
[  4]   1.00-2.00   sec  1.08 GBytes  9.29 Gbits/sec  455    724 KBytes
[  4]   2.00-3.00   sec  1.08 GBytes  9.31 Gbits/sec    0   1.40 MBytes
[  4]   3.00-4.00   sec  1.07 GBytes  9.23 Gbits/sec  1335    567 KBytes
[  4]   4.00-5.00   sec  1.08 GBytes  9.31 Gbits/sec  313    936 KBytes
[  4]   5.00-6.00   sec  1.08 GBytes  9.30 Gbits/sec  447    643 KBytes
[  4]   6.00-7.00   sec  1.08 GBytes  9.28 Gbits/sec  611    752 KBytes
[  4]   7.00-8.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.40 MBytes
[  4]   8.00-9.00   sec  1.08 GBytes  9.27 Gbits/sec  222    624 KBytes
[  4]   9.00-10.00  sec  1.08 GBytes  9.31 Gbits/sec  547   1.11 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec  4141             sender
[  4]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec                  receiver

Looks like it is Docker. I'll dig deeper.

@wkruse
Copy link
Author

wkruse commented Jun 24, 2020

@lucab I was testing with and without teaming as I created the issue, it didn't make any difference. Also no differences on ip link show <IFACE> besides teaming/bonding

Fedora CoreOS

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP mode DEFAULT group default qlen 1000
    link/ether
eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP mode DEFAULT group default qlen 1000
    link/ether

CoreOS

eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether
eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether

@wkruse
Copy link
Author

wkruse commented Jun 24, 2020

Docker version 19.03.8 (Fedora CoreOS) vs 1.12.6 (CoreOS).

@bgilbert
Copy link
Contributor

@wkruse Out of curiosity, do you also see this problem on Container Linux if you write no to /etc/coreos/docker-1.12 and reboot?

@wkruse
Copy link
Author

wkruse commented Jun 25, 2020

@bgilbert There is sadly no CoreOS release, that supports Docker version 19.03.8.

@bgilbert
Copy link
Contributor

No, but you could try with 18.06.3.

@wkruse
Copy link
Author

wkruse commented Jun 25, 2020

@bgilbert CoreOS 2512.3.0 with Docker 18.06.3 running iperf3 with network virtualization is fluctuating between 6-8 Gbits/sec

Connecting to host xx.xx.xx.xx, port 5201
[  4] local 172.17.0.2 port 43954 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   631 MBytes  5.29 Gbits/sec  932    853 KBytes
[  4]   1.00-2.00   sec   946 MBytes  7.94 Gbits/sec   80   1.17 MBytes
[  4]   2.00-3.00   sec   961 MBytes  8.07 Gbits/sec  148   1.38 MBytes
[  4]   3.00-4.00   sec   826 MBytes  6.93 Gbits/sec  435    922 KBytes
[  4]   4.00-5.00   sec   908 MBytes  7.61 Gbits/sec  261   1.05 MBytes
[  4]   5.00-6.00   sec   866 MBytes  7.27 Gbits/sec  245    841 KBytes
[  4]   6.00-7.00   sec   925 MBytes  7.76 Gbits/sec  116   1.25 MBytes
[  4]   7.00-8.00   sec   910 MBytes  7.63 Gbits/sec  459    571 KBytes
[  4]   8.00-9.00   sec   875 MBytes  7.34 Gbits/sec    3   1.19 MBytes
[  4]   9.00-10.00  sec   916 MBytes  7.68 Gbits/sec  108   1.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.56 GBytes  7.35 Gbits/sec  2787             sender
[  4]   0.00-10.00  sec  8.56 GBytes  7.35 Gbits/sec                  receiver

iperf Done.

with --net=host it is 9.29 Gbits/sec.

So I guess it is Docker and not Fedora CoreOS.

@wkruse wkruse mentioned this issue Jun 26, 2020
Closed
3 tasks
@wkruse
Copy link
Author

wkruse commented Aug 21, 2020

The solution for me is not to use Docker network virtualization. Closing it.

@wkruse wkruse closed this as completed Aug 21, 2020
@wkruse wkruse mentioned this issue Mar 12, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lucab @cgwalters @bgilbert @wkruse @jlebon