docker deamon not restarted after upgrade

[gilligan]

Issue description

I ran an a deployment that updated docker from 1.10.3 to 1.12.1. After the deployment I noticed that I was running into an error:

10:43:09 Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.22)

It turned out that the old docker deamon version was still running. After running systemctl restart docker the correct docker deamon version was running. Oddly enough I also had to do systemctl restart docker.socket because the socket was missing.

Steps to reproduce

Not sure how to reproduce this.. i updated an existing nixops controlled server to 16.09 and the above happened.

Technical details

• System: 16.09pre-git (Flounder)
• Nix version: 1.11.4
• Nixpkgs version: 15.09.505.7adab11

[NeQuissimus]

I think I may have seen this when changing my docker configuration and running nixos-rebuild switch (adding a private repository).
I don't think we manage the socket and restarting docker does not kill the socket.
I am unsure whether this can be considered an upstream bug or not.
Maybe somebody with more in-depth Docker experience can look into this?

[gilligan]

Actually I should add something to this. After upgrading I ran into some problems with dockerTools (#19477) so I reverted my update to get the old docker version again. That lead to a series of problems and eventually to me destroying and redeploying my server:

After switching back to a previous nixpkgs set with docker 1.10.3 I could not start the docker deamon anymore:

Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.230373069Z" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production
Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.245786593Z" level=warning msg="devmapper: Base device already exists and has filesystem ext4 on it. User sp
Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.277894821Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.280269281Z" level=info msg="Firewalld running: false"
Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.331406661Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon o
Oct 12 10:48:23 jenkinsMaster docker[49224]: time="2016-10-12T10:48:23.409050730Z" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating d
Oct 12 10:48:23 jenkinsMaster systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Oct 12 10:48:23 jenkinsMaster systemd[1]: docker.service: Unit entered failed state.
Oct 12 10:48:23 jenkinsMaster systemd[1]: docker.service: Failed with result 'exit-code'.

So if I am reading this correctly the deamon failed to start because the docker0 interface was there already. I tried removing the network interface and starting docker again but this had the same result.

I then disabled docker and deployed again. To my surprise the docker0 interface was still there. Deploying with docker enabled again led to the same error as above.

I gave up and destroyed the server and deployed again ;/

[Mic92]

Looks like not restarting is intended: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/docker.nix#L120

This could be reconsidered, because newer docker version can support restarting dockerd without restarting containers.

[Mic92]

The second error you reported here is most likely because two instances running at the same time.

[aborsu]

@Mic92 I'm testing making restarting or not an option here

If it solves the problem for me and I don't notice any other problem I'll open a pull-request

[aborsu]

@NeQuissimus
Seems to me that this issue can be closed thanks to #21326