running bash from host to container is very slow

[leah4kosh]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one
  I'm downloading a large file from aws s3, usnig the asw cli. when running inside the docker terminal, the file is downloaded in 4 minutes.
  If I execute the command from the host to the docker, i.e.
  docker exec -i <container_id> /bin/bash -lc "aws s3..." it will take almost 40 minutes for the same command.

I'm running on Ubuntu 16.04, docker-ce 18.06, my docker image is based ubuntu 16.04 as well
What would be the reason for such behavior?

Expected behavior

The same time should take form the host to the container as from the container's terminal

Actual behavior

It takes 4 minutes from the docker's terminal and 40 minutes from the host to the docker

Steps to reproduce the behavior

create an ubuntu docker image with aws cli installed
create a docker container and connect to it (i.e. docker exec -it container_id /bin/bash -l)
run aws s3 .. - to download a file from the containers terminal
exit the container and run: docker exec -i container_id /bin/bash -lc "aws s3.."
it will take almost 10 times the time.

Output of docker version:

docker-ce 18.06

[thaJeztah]

Could you add the full output of docker version and docker info?

Is this only when downloading from s3, or do you see the same when downloading from (eg) a large file from a web server (using, e.g. curl)?

Probably important to try to narrow down the bottleneck; is it networking speed that's slow, or is the bottleneck in writing the file to the container's filesystem?

Does speed improve if you use a volume to write the file to (circumventing the container's copy-on-write filesystem)?

[benjah1]

I encounter a similar problem. See the example below,

#!/bin/bash

docker run -d --rm --name test alpine sleep 10000

for I in {1..10}
do
  # docker exec test sh -c "echo this is ${I}"
  sh -c "echo this is ${I}"
done

docker stop test

The docker exec is way much slower than the native sh. The become a serious performance issue when a large amounts of commands need to be executed.

My real life use case is using Ansible Molecule to develop ansible role. Test cases are running very slowly with docker connections.

[cpuguy83]

You could just do it all in one exec session?
Exec will always be slower than native because there's a lot of things happening.

You say it's much slower, how much slower?

[benjah1]

#!/bin/bash

# docker run -d --rm --name test alpine sleep 10000 # pre-run

for I in {1..100}
do
  # docker exec test sh -c "echo this is ${I}"
  sh -c "echo this is ${I}"
done

# docker stop test # post-run

With native,

$ time bash ./test.sh
this is 1
...
this is 100
bash ./test.sh  0.04s user 0.03s system 85% cpu 0.082 total

With docker exec

$ time bash ./test.sh
this is 1
...
this is 100
bash ./test.sh  2.23s user 2.57s system 10% cpu 44.423 total

Unfortunately, the framework(testinfra) is build to exec command with docker exec one by one. It may be possible to modify the connector in Ansible or pytest(testinfra) to improve the performance. But still, I think docker exec is the root problem, and should be improved if possible.

[cpuguy83]

There is a known issue with the newer version of libseccomp that could be causing a slowdown here.
What version of the libseccomp2 package do you have installed? What distro and docker version?

[benjah1]

I am testing it on my home laptop now, it is,
docker: 17.11.0-ce
libseccomp2: 2.4.1-0ubuntu0.16.04.2
Ubuntu 16.04 LTS by Virtualbox on Windows 10

Upgrade to docker: 19.03.2, it is still the same.

This is also happening in centos72 on aws ec2 with an older version of docker.
and Ubuntu 18.04 (by Virtualbox) on OSX (macbook pro 13 2018).

[cpuguy83]

For reference: seccomp/libseccomp#153 libseccomp 2.4 seems like a possible cause of this slow down. Could you downgrade to see if that improves things? note that there is a publicly disclosed vulnerability in older versions of libseccomp as mentioned in the referenced issue :(

…

On Sat, Sep 7, 2019 at 7:22 AM Ben Huang ***@***.***> wrote: I am testing it on my home laptop now, it is, docker: 17.11.0-ce libseccomp2: 2.4.1-0ubuntu0.16.04.2 Ubuntu 16.04 LTS by Virtualbox on Windows 10 Upgrade to docker: 19.03.2, it is still the same. This is also happening in centos72 on aws ec2 with an older version of docker. and Ubuntu 18.04 (by Virtualbox) on OSX (macbook pro 13 2018). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#466?email_source=notifications&email_token=AAGDCZQYZM5R6L47NMTUP5TQIO2JPA5CNFSM4F7K2DL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6EZ3XQ#issuecomment-529112542>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGDCZVTDHEXHTCE7ZQBKV3QIO2JPANCNFSM4F7K2DLQ> .

-- - Brian Goff

[benjah1]

I downgraded docker to 17.12.0-ce, libseccomp2 2.2.3-3ubuntu3.

$ time bash ./test.sh
this is 1
...
this is 100
bash ./test.sh  2.51s user 1.48s system 23% cpu 17.079 total

With native,

$ time bash ./test.sh
this is 1
...
this is 100
bash ./test.sh  0.02s user 0.01s system 13% cpu 0.203 total

libseccomp2 2.2 is much faster than 2.4. However, native is still much faster...

[cpuguy83]

I definitely expect startup to be significantly slower than native since it is doing tons of extra things.
There's (multiple) API requests over a unix socket and lots of extra setup for the process itself such as applying a seccomp profile, setting namespaces, dropping privileges, etc.