None w/ Docker 18.09: [ERROR SystemVerification]: unsupported docker version: 18.09.0

[Ark-kun]

This is a BUG REPORT

VM-less Minikube fails to start on Docker 18.09:

[ERROR SystemVerification]: unsupported docker version: 18.09.0

Environment:
Minikube version: v1.12.2

• OS: Debian 9
• VM Driver : None

What happened:
Our test script that installs Docker and VM-less Minikube started failing. Upgrading Minikube to 1.12.2 did not help.

**Logs:

+ export MINIKUBE_WANTUPDATENOTIFICATION=false
+ MINIKUBE_WANTUPDATENOTIFICATION=false
+ export MINIKUBE_WANTREPORTERRORPROMPT=false
+ MINIKUBE_WANTREPORTERRORPROMPT=false
+ export MINIKUBE_HOME=/home/not-root
+ MINIKUBE_HOME=/home/not-root
+ export CHANGE_MINIKUBE_NONE_USER=true
+ CHANGE_MINIKUBE_NONE_USER=true
+ mkdir -p /home/not-root/.kube
+ touch /home/not-root/.kube/config
+ export KUBECONFIG=/home/not-root/.kube/config
+ KUBECONFIG=/home/not-root/.kube/config
+ sudo -E /usr/local/bin/minikube start --vm-driver=none --kubernetes-version=v1.12.2
Starting local Kubernetes v1.12.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.12.2
Downloading kubelet v1.12.2
Finished Downloading kubeadm v1.12.2
Finished Downloading kubelet v1.12.2
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E1109 23:47:22.788065   12517 start.go:297] Error starting cluster:  kubeadm init error 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns
 running command: : running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns

 output: [init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
	[WARNING FileExisting-ebtables]: ebtables not found in system path
	[WARNING FileExisting-socat]: socat not found in system path
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.9.0-8-amd64
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled (as module)
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
DOCKER_VERSION: 18.09.0
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
	[WARNING Hostname]: hostname "minikube" could not be reached
	[WARNING Hostname]: hostname "minikube" lookup minikube on 169.254.169.254:53: no such host
[preflight] Some fatal errors occurred:
	[ERROR SystemVerification]: unsupported docker version: 18.09.0
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
: running command: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon coredns

[ryanolson]

Confirmed. I see the same.

[afbjorklund]

You need to downgrade docker to 18.06, until 18.09 is validated and supported by kubeadm:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/docker_validator.go#L41

It was only recently that kubeadm started to support anything beyond version 17.03: #3223

Apparently you are also missing other binaries, such as socat and maybe also crictl...

[afbjorklund]

See https://kubernetes.io/docs/setup/cri/#docker for instructions. Or just use minikube.iso.

Not sure if there was a bug report filed with kubernetes/kubeadm already, that would be next.

[thomas-riccardi]

@afbjorklund kubernetes pre-flight checks have 3 possible outcome: OK, WARNING, ERROR.

The docker version check can return the 3 outcomes:

• supported version: OK
• unsupported but recognized version: WARNING
• all others: ERROR

Since docker 18.09 we get an ERROR instead of an expected WARNING.

Analysis

Code

https://github.com/kubernetes/kubernetes/blame/master/cmd/kubeadm/app/util/system/docker_validator.go#L73-L99

• matched is for supported versions: in our case it's false
• then it checks for recognized version with \d{2}\.\d+\.\d+-[a-z]{2} regex

Expected result:
the ver regex matches on the 18.09 docker version, and thus returns a WARNING.

Actual result:
the ver regex does no match, and thus returns an ERROR.

Docker versions

18.06

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

18.09

$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

They dropped the -ce suffix in this new version => the regex should be updated.

[afbjorklund]

Nice analysis! But still needs a kubeadm update ? Both to fix the regexp, and to validate if the new version "works"...

[tstromberg]

It's worth noting that our CI "None" instance recently upgraded docker to 18.09, so it's failing tests due to this issue as well.

[afbjorklund]

@tstromberg : according to the kubeadm documentation, it is supposed to be on "hold" for now:

## Install docker.
apt-get update && apt-get install docker-ce=18.06.0~ce~3-0~ubuntu

## Install docker.
yum update && yum install docker-ce-18.06.1.ce

https://kubernetes.io/docs/setup/cri/#docker

I also dislike the way that the new "containerd.io" package now conflicts with "runc", breaking apt...

Unpacking containerd.io (1.2.0~rc.2-1) ...
dpkg: error processing archive /var/cache/apt/archives/containerd.io_1.2.0~rc.2-1_amd64.deb (--unpack):
 trying to overwrite '/usr/sbin/runc', which is also in package runc 1.0.0~rc2+docker1.13.1-0ubuntu1~16.04.1
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Then again "cri-o-1.10" uses "cri-o-runc", so I guess they can coexist. Just bad upgrade experience.

[balopat]

it's helpful to get the exact docker-ce version number on non-ubuntu debian versions:

apt-cache madison docker-ce | grep 18.06 | awk '{print $3}' | head -n 1

[balopat]

I downgraded the CI server so None driver tests are passing again.
Even though when 1.13 comes out we have to think about this a little bit: do we make 1.13 the default for None integration tests? Or we wait with the docker upgrade until 1.13 becomes the default in minikube?

[tsuna]

Since the fix in kubernetes/kubernetes#71001 merely consists in updating the regexp to ignore the -ce suffix, is there a known workaround we can use in the meantime?

[stucki]

How about this:

kubeadm init [...] --ignore-preflight-errors=SystemVerification

[afbjorklund]

Why not just use the older Docker version ?

[stucki]

If I had known before the Docker update, then yes. Didn't know about it and don't like to downgrade.

[gatsbys]

Removing the old version with :
sudo apt-get purge docker-ce
sudo rm -rf /var/lib/docker

And then reinstall as @afbjorklund said :

apt-get update && apt-get install docker-ce=18.06.0~ce~3-0~ubuntu

Worked for me.

[aashish-sheshadri]

Is there a way to add --ignore-preflight-errors=SystemVerification?

[srikiz]

The above comments from @gatsbys worked for me perfectly.

[eltsdev]

Happening to me too! I confirm that latest docker version (Docker 18.09) is not supported by minikube using no vm option in Ubuntu 16.04.

[tstromberg]

I haven't confirmed yet, but I think this should be resolved by using a newer Kubernetes release (v1.13.x) according to https://kubernetes.io/docs/setup/release/notes/

If you update to minikube v0.33.0, it defaults to Kubernetes v1.13.2, though you can also select it using --kubernetes-version in v0.32.0. Does someone mind confirming whether this is an issue any longer?

It would be nice to allow users to ignore SystemVerification errors. I've opened #3543 to address that feature request.

[thomas-riccardi]

@tstromberg I can confirm this now works with the following setup:

• docker 18.09.0 or 18.09.1
• minikube 0.32.0
• --kuberentes-version 1.31.1 or 1.13.2

So this should work again out-of-the-box with minikube 0.33.0 which defaults to kubernetes 1.13.2.

[dlakatos847]

@tstromberg confirmed, works out of the box 👍

• OpenSuSE Tumbleweed x86-64 up-to-date at 2019.02.13. running on Linux kernel 4.20.6
• Docker CE 18.09
• Minikube v0.33.1
• Kubernetes v1.13.2

[marcosdiez]

if you are using ubuntu 16.04, this is the magic command that solves it all
sudo apt-get install docker.io=18.06.1-0ubuntu1.2~16.04.1

[dperique]

@marcosdiez yes, that worked for me. I had to install that and answered to the part where it asks if you want to restart docker now. I then restarted minikube and it seems happy again.

Thanks!

[marcosdiez]

For those who do not want to downgrade docker, I sent this PR #3879 which will allow us to pass a parameter to minikube to ignore docker changes.

After the PR gets merged, this will be the command to run sudo -E minikube start --vm-driver=none --kubernetes-version v1.11.8 --extra-config kubeadm.ignore-preflight-errors=SystemVerification

As of v0.35.0 this has not been merged.

[afbjorklund]

Most likely minikube will support and use the same versions of Docker that kubeadm does:

https://kubernetes.io/docs/setup/cri/#docker

Version 18.06.2 is recommended, but 1.11, 1.12, 1.13, 17.03 and 18.09 are known to work as well. Keep track of the latest verified Docker version in the Kubernetes release notes.

Docker 18.09 is supported since Kubernetes 1.13, This is mostly an issue with none driver.