Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Network performance and bandwidth degradation #1048

Closed
2 of 3 tasks
wkruse opened this issue Jun 26, 2020 · 9 comments
Closed
2 of 3 tasks

Network performance and bandwidth degradation #1048

wkruse opened this issue Jun 26, 2020 · 9 comments

Comments

@wkruse
Copy link

wkruse commented Jun 26, 2020

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

CoreOS 1632.3.0 with Docker 1.12.6 running iperf3 on two machines in containers with network virtualization, we are getting 9.23 Gbits/sec

Connecting to host xx.xx.xx.xx, port 5201
[  4] local 172.17.0.2 port 56838 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.07 GBytes  9.23 Gbits/sec  104    686 KBytes
[  4]   1.00-2.00   sec  1.07 GBytes  9.20 Gbits/sec   32    683 KBytes
[  4]   2.00-3.00   sec  1.07 GBytes  9.19 Gbits/sec   43    609 KBytes
[  4]   3.00-4.00   sec  1.07 GBytes  9.23 Gbits/sec   38    769 KBytes
[  4]   4.00-5.00   sec  1.08 GBytes  9.25 Gbits/sec   23    691 KBytes
[  4]   5.00-6.00   sec  1.07 GBytes  9.23 Gbits/sec   25    659 KBytes
[  4]   6.00-7.00   sec  1.07 GBytes  9.23 Gbits/sec   45    798 KBytes
[  4]   7.00-8.00   sec  1.08 GBytes  9.24 Gbits/sec   16    649 KBytes
[  4]   8.00-9.00   sec  1.08 GBytes  9.24 Gbits/sec   36    725 KBytes
[  4]   9.00-10.00  sec  1.07 GBytes  9.23 Gbits/sec   56    857 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.7 GBytes  9.23 Gbits/sec  418             sender
[  4]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec                  receiver

iperf Done.

also tested with qperf

tcp_bw:
    bw              =   1.14 GB/sec
    msg_rate        =   17.4 K/sec
    port            =  4,001
    time            =     60 sec
    send_cost       =    278 ms/GB
    recv_cost       =    630 ms/GB
    send_cpus_used  =   31.6 % cpus
    recv_cpus_used  =   71.6 % cpus
tcp_lat:
    latency        =     24 us
    msg_rate       =   41.7 K/sec
    port           =  4,001
    time           =     60 sec
    loc_cpus_used  =   36.1 % cpus
    rem_cpus_used  =   48.8 % cpus

I would expect similar results in Docker 19.03.8.

Actual behavior

Fedora CoreOS 32.20200615.1.3 with Docker 19.03.8 running iperf3 on two machines in containers with network virtualization is about the half of the bandwidth

Connecting to host xx.xx.xx.xx, port 5201
[  4] local 172.17.0.2 port 51522 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   733 MBytes  6.15 Gbits/sec  1567    529 KBytes
[  4]   1.00-2.00   sec   599 MBytes  5.02 Gbits/sec  841    608 KBytes
[  4]   2.00-3.00   sec   629 MBytes  5.27 Gbits/sec  1655    631 KBytes
[  4]   3.00-4.00   sec   609 MBytes  5.11 Gbits/sec  1024    689 KBytes
[  4]   4.00-5.00   sec   591 MBytes  4.96 Gbits/sec  597    600 KBytes
[  4]   5.00-6.00   sec   599 MBytes  5.02 Gbits/sec  1214    437 KBytes
[  4]   6.00-7.00   sec   599 MBytes  5.02 Gbits/sec   55   1.00 MBytes
[  4]   7.00-8.00   sec   664 MBytes  5.57 Gbits/sec  1441    723 KBytes
[  4]   8.00-9.00   sec   591 MBytes  4.96 Gbits/sec  528   1.02 MBytes
[  4]   9.00-10.00  sec   654 MBytes  5.48 Gbits/sec  742    891 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  6.12 GBytes  5.26 Gbits/sec  9664             sender
[  4]   0.00-10.00  sec  6.12 GBytes  5.25 Gbits/sec                  receiver

iperf Done.

also tested with qperf

tcp_bw:
    bw              =    652 MB/sec
    msg_rate        =    9.9 K/sec
    port            =  4,001
    time            =     60 sec
    send_cost       =   2.17 sec/GB
    recv_cost       =   3.27 sec/GB
    send_cpus_used  =    142 % cpus
    recv_cpus_used  =    214 % cpus
tcp_lat:
    latency        =   39.9 us
    msg_rate       =   25.1 K/sec
    port           =  4,001
    time           =     60 sec
    loc_cpus_used  =    103 % cpus
    rem_cpus_used  =    122 % cpus

Steps to reproduce the behavior

Running iperf3 as server on one machine

docker run  -it --rm --name=iperf3-server -p 5201:5201 networkstatic/iperf3 -s

Running iperf3 as client on the second machine

docker run  -it --rm networkstatic/iperf3 -c xx.xx.xx.xx

Running qperf as server on one machine

docker run -d --rm --name=qperf-server -p 4000:4000 -p 4001:4001 xridge/qperf -lp 4000

Running qperf as client on the second machine

docker run -it --rm xridge/qperf -t 60 -v xx.xx.xx.xx -lp 4000 -ip 4001 tcp_bw tcp_lat

Output of docker version:

CoreOS 1632.3.0

 $ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.6
 Git commit:   d9ad3fc
 Built:        Wed Feb 14 03:24:39 2018
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.6
 Git commit:   d9ad3fc
 Built:        Wed Feb 14 03:24:39 2018
 OS/Arch:      linux/amd64

Fedora CoreOS 32.20200615.1.3

 $  docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.14.2
 Git commit:        afacb8b
 Built:             Thu May  7 18:59:17 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.2
  Git commit:       afacb8b
  Built:            Thu May  7 00:00:00 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3
  GitCommit:
 runc:
  Version:          1.0.0-rc10+dev
  GitCommit:        fbdbaf85ecbc0e077f336c03062710435607dbf1
 docker-init:
  Version:          0.18.0
  GitCommit:

Output of docker info:

CoreOS 1632.3.0

$ docker info
Containers: 68
 Running: 67
 Paused: 0
 Stopped: 1
Images: 414
Server Version: 1.12.6
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp selinux
Kernel Version: 4.14.19-coreos
Operating System: Container Linux by CoreOS 1632.3.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.88 GiB
Name: xxx
ID: xxx
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false

Fedora CoreOS 32.20200615.1.3

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 73
  Running: 67
  Paused: 0
  Stopped: 6
 Images: 15
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: /usr/libexec/docker/docker-init
 containerd version:
 runc version: fbdbaf85ecbc0e077f336c03062710435607dbf1
 init version:
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 5.6.18-300.fc32.x86_64
 Operating System: Fedora CoreOS 32.20200615.1.3
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 62.8GiB
 Name: xxx
 ID: xxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.)

Bare metal install on Dell PowerEdge R630 server.

Fedora CoreOS 32.20200615.1.3 with Docker 19.03.8, when running the tests with --net=host we are getting the expected results.

Connecting to host xx.xx.xx.xx, port 5201
[  4] local xx.xx.xx.xx port 34754 connected to xx.xx.xx.xx port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.07 GBytes  9.22 Gbits/sec  168   1.08 MBytes
[  4]   1.00-2.00   sec  1.09 GBytes  9.35 Gbits/sec   14   1.63 MBytes
[  4]   2.00-3.00   sec  1.07 GBytes  9.17 Gbits/sec  286    956 KBytes
[  4]   3.00-4.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.59 MBytes
[  4]   4.00-5.00   sec  1.09 GBytes  9.33 Gbits/sec   35    943 KBytes
[  4]   5.00-6.00   sec  1.09 GBytes  9.32 Gbits/sec   12    759 KBytes
[  4]   6.00-7.00   sec  1.09 GBytes  9.33 Gbits/sec   23    947 KBytes
[  4]   7.00-8.00   sec  1.09 GBytes  9.34 Gbits/sec   10   1.02 MBytes
[  4]   8.00-9.00   sec  1.09 GBytes  9.32 Gbits/sec   26    611 KBytes
[  4]   9.00-10.00  sec  1.09 GBytes  9.32 Gbits/sec    2   1.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec  576             sender
[  4]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec                  receiver

iperf Done.

tcp_bw:
    bw              =   1.16 GB/sec
    msg_rate        =   17.6 K/sec
    port            =  4,001
    time            =     60 sec
    send_cost       =    827 ms/GB
    recv_cost       =   1.52 sec/GB
    send_cpus_used  =   95.5 % cpus
    recv_cpus_used  =    176 % cpus
tcp_lat:
    latency        =   31.2 us
    msg_rate       =     32 K/sec
    port           =  4,001
    time           =     60 sec
    loc_cpus_used  =   79.8 % cpus
    rem_cpus_used  =    116 % cpus
@wkruse
Copy link
Author

wkruse commented Jun 26, 2020

Related to coreos/fedora-coreos-tracker/issues/542

@thaJeztah
Copy link
Member

@arkodg ptal

@wkruse
Copy link
Author

wkruse commented Jul 3, 2020

I've tried disabling seccomp and AppArmor with --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined and re-running the tests with no luck.

@wkruse
Copy link
Author

wkruse commented Jul 3, 2020

Also additionally disabling SELinux by removing --selinux-enabled from /etc/sysconfig/docker, setting SELINUX=disabled in /etc/selinux/config, rebooting and then running the containers with --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --security-opt label:disable didn't help.

@arkodg
Copy link

arkodg commented Jul 3, 2020

can you try increasing the Bridge MTU

@wkruse
Copy link
Author

wkruse commented Jul 3, 2020

The default MTU is 1500 on the host and also for docker0. I've added --mtu=9000 to DOCKER_OPTS, systemctl daemon-reload, systemctl restart docker, but in ip adds show the bridge still had 1500.

ifconfig docker0 mtu 9000 did the trick.

 docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc noqueue state DOWN group default
    link/ether 02:42:18:5b:4f:86 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:18ff:fe5b:4f86/64 scope link
       valid_lft forever preferred_lft forever

Still getting the same results, no change.

@wkruse
Copy link
Author

wkruse commented Jul 10, 2020

@arkodg What would be a rule for the bridge MTU size? Also, when we compare the number of retransmits above, it is about 10.000 with network virtualization in Docker 19.03.8 vs 400 in Docker 1.12.6 vs 600 without (--net=host) in Docker 19.03.8. Could that be a hint?

@arkodg
Copy link

arkodg commented Jul 10, 2020

the one intermediate layer that can play a factor here is conntrack (connection tracking module for NATing). You could try further debugging the state of conntrack

@wkruse
Copy link
Author

wkruse commented Aug 21, 2020

The solution for me is not to use Docker network virtualization. Closing it.

@wkruse wkruse closed this as completed Aug 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thaJeztah @wkruse @arkodg