Docker network / socket issues #11478

jgillich · 2015-12-05T03:37:29Z

Two issues here.

Docker networking doesn't work after boot (before I upgraded to unstable, restarting docker fixed that. --net host works too)
systemctl restart docker doesn't recreate the socket /var/run/docker.sock

So here is bascially what happens:

$ docker run ...
Error response from daemon: 404 page not found

$ sudo systemctl restart docker

$ docker run ...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

$ sudo file /var/run/docker.sock
/var/run/docker.sock: cannot open `/var/run/docker.sock' (No such file or directory)

Journal logs:

level=warning msg="Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
level=warning msg="Running modprobe nf_nat failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
level=info msg="Firewalld running: false"
level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
level=warning msg="Your kernel does not support kernel memory limit."
level=warning msg="Your kernel does not support cgroup cfs period"
level=warning msg="Your kernel does not support cgroup cfs quotas"
level=info msg="Loading containers: start."
level=info msg="Loading containers: done."
level=info msg="Daemon has completed initialization"
level=info msg="Docker daemon" commit=76d6bc9a execdriver=native-0.2 graphdriver=devicemapper version=1.9.0
level=info msg="Processing signal 'terminated'"
level=info msg="API listen on /var/run/docker.sock"
level=warning msg="Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
level=warning msg="Running modprobe nf_nat failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
level=info msg="Firewalld running: false"
level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.1/16. Daemon option --bip can be used to set a preferred IP address"
level=warning msg="Your kernel does not support kernel memory limit."
level=warning msg="Your kernel does not support cgroup cfs period"
level=warning msg="Your kernel does not support cgroup cfs quotas"
level=info msg="Loading containers: start."
level=info msg="Loading containers: done."
level=info msg="Daemon has completed initialization"
level=info msg="Docker daemon" commit=76d6bc9a execdriver=native-0.2 graphdriver=devicemapper version=1.9.0

The text was updated successfully, but these errors were encountered:

jgillich · 2015-12-07T16:50:52Z

The networking issue on boot seems to be fixed now, but still no socket after restart.

jgillich · 2015-12-07T17:06:06Z

Setting socketActivation = false fixes it. With it enabled, I get this in the logs:

Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.275389920+01:00" level=warning msg="Your kernel does not support kernel memory limit."
Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.275565255+01:00" level=warning msg="Your kernel does not support cgroup cfs period"
Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.275603085+01:00" level=warning msg="Your kernel does not support cgroup cfs quotas"
Dec 07 18:02:44 thinkpad systemd[1]: libcontainer-4948-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.281643862+01:00" level=info msg="Loading containers: start."
Dec 07 18:02:44 thinkpad docker[4948]: ..........................................................................................................................................................................
Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.324232824+01:00" level=info msg="Loading containers: done."
Dec 07 18:02:44 thinkpad docker[4948]: time="2015-12-07T18:02:44.324258519+01:00" level=info msg="Daemon has completed initialization"

Wondering if the Scope has no PIDs. Refusing. error is related.
@lethalman

domenkozar · 2015-12-11T18:38:32Z

@jgillich you mean socketActivation = true? That helped me.

jgillich · 2015-12-12T05:19:27Z

@domenkozar It's true by default, I had to set it to false.

jgeerds · 2015-12-23T10:21:00Z

Disabling socketActivation fixes the issue for me too

chris-martin · 2016-02-14T12:11:27Z

Likewise, I just moved from 15.09 to unstable, and had to set virtualisation.docker.socketActivation = false; to get Docker working again.

offlinehacker · 2016-02-14T20:22:03Z

Well, in my case it just takes a few moments(~20seconds) to start-up, you
can just wait a bit and it will start.

On Sun, Feb 14, 2016 at 1:11 PM Christopher Martin notifications@github.com
wrote:

Likewise, I just moved from 15.09 to unstable, and had to set virtualisation.docker.socketActivation
= false; to get Docker working again.

—
Reply to this email directly or view it on GitHub
#11478 (comment).

jgeerds · 2016-02-16T15:50:33Z

@offlinehacker even after five minutes socketActivation = true doesn't work for me

ibrahimsag · 2016-12-20T12:01:23Z

@jgillich any updates?

jgillich · 2016-12-20T12:22:55Z

No, I'm not even using NixOS currently. But feel free to look into it yourself.

ibrahimsag · 2016-12-20T12:24:55Z

i've solved this by systemctl restart docker.socket. we can close this issue then.

jgillich · 2016-12-20T12:40:23Z

No, that's no a solution. FWIW, there's a similar issue with socket activation on Ubuntu Xenial: moby/moby#23793

It causes issues for many people, see NixOS#11478

All the new options in detail: Enable docker in multi-user.target make container created with restart=always to start. We still want socket activation as it decouples dependencies between the existing of /var/run/docker.sock and the docker daemon. This means that services can rely on the availability of this socket. Fixes NixOS#11478 NixOS#21303 wantedBy = ["multi-user.target"]; This allows us to remove the postStart hack, as docker reports on its own when it is ready. Type=notify The following will set unset some limits because overhead in kernel's ressource accounting was observed. Note that these limit only apply to containerd. Containers will have their own limit set. LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Upgrades may require schema migrations. This can delay the startup of dockerd. TimeoutStartSec=0 Allows docker to create its own cgroup subhierarchy to apply ressource limits on containers. Delegate=true When dockerd is killed, container should be not affected to allow `live restore` to work. KillMode=process

Mic92 · 2016-12-21T10:13:11Z

docker is also missing the modprobe binary - usually such firewall modules are automatically loaded by iptables but maybe we could add the binary to the paths to silence the warning.

All the new options in detail: Enable docker in multi-user.target make container created with restart=always to start. We still want socket activation as it decouples dependencies between the existing of /var/run/docker.sock and the docker daemon. This means that services can rely on the availability of this socket. Fixes #11478 #21303 wantedBy = ["multi-user.target"]; This allows us to remove the postStart hack, as docker reports on its own when it is ready. Type=notify The following will set unset some limits because overhead in kernel's ressource accounting was observed. Note that these limit only apply to containerd. Containers will have their own limit set. LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Upgrades may require schema migrations. This can delay the startup of dockerd. TimeoutStartSec=0 Allows docker to create its own cgroup subhierarchy to apply ressource limits on containers. Delegate=true When dockerd is killed, container should be not affected to allow `live restore` to work. KillMode=process

jgillich mentioned this issue Dec 10, 2015

rkt: add service #11565

Merged

jgillich mentioned this issue Mar 25, 2016

Docker container can't connect to internet #14200

Closed

jgillich added a commit to jgillich/nixpkgs that referenced this issue Dec 20, 2016

docker: disable socket activation by default

5ec5130

It causes issues for many people, see NixOS#11478

jgillich mentioned this issue Dec 20, 2016

docker: disable socket activation by default #21303

Closed

Mic92 mentioned this issue Dec 21, 2016

docker: update service units from upstream #21326

Merged

3 tasks

Mic92 closed this as completed in #21326 Dec 23, 2016

codygman mentioned this issue Mar 23, 2021

Docker builds stopped working after update divnix/digga#209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker network / socket issues #11478

Docker network / socket issues #11478

jgillich commented Dec 5, 2015

jgillich commented Dec 7, 2015

jgillich commented Dec 7, 2015

domenkozar commented Dec 11, 2015

jgillich commented Dec 12, 2015

jgeerds commented Dec 23, 2015

chris-martin commented Feb 14, 2016

offlinehacker commented Feb 14, 2016

jgeerds commented Feb 16, 2016

ibrahimsag commented Dec 20, 2016

jgillich commented Dec 20, 2016

ibrahimsag commented Dec 20, 2016

jgillich commented Dec 20, 2016 •

edited

Loading

Mic92 commented Dec 21, 2016

Docker network / socket issues #11478

Docker network / socket issues #11478

Comments

jgillich commented Dec 5, 2015

jgillich commented Dec 7, 2015

jgillich commented Dec 7, 2015

domenkozar commented Dec 11, 2015

jgillich commented Dec 12, 2015

jgeerds commented Dec 23, 2015

chris-martin commented Feb 14, 2016

offlinehacker commented Feb 14, 2016

jgeerds commented Feb 16, 2016

ibrahimsag commented Dec 20, 2016

jgillich commented Dec 20, 2016

ibrahimsag commented Dec 20, 2016

jgillich commented Dec 20, 2016 • edited Loading

Mic92 commented Dec 21, 2016

jgillich commented Dec 20, 2016 •

edited

Loading