"kubeadm init" fails when run offline due to "kubadm config images list" returning an incorrect list of images

[mlgibbons]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version:
  Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Windows 10 running minikube

• OS (e.g. from /etc/os-release):
  NAME=Buildroot
  VERSION=2018.05
  ID=buildroot
  VERSION_ID=2018.05
  PRETTY_NAME="Buildroot 2018.05"

• Kernel (e.g. uname -a):
  Linux test-dis2 4.15.0 kubeadm join on slave node fails preflight checks #1 SMP Fri Oct 5 20:44:14 UTC 2018 x86_64 GNU/Linux

What happened?

When trying to start up a minikube instance for K8S v1.11.0 the process hangs if the operation is performed offline.

This initially appeared to be due to minikube having not previously cached all of the required images.

I then ran "kubeadm config images list --kubernetes-version=v1.11.0" to get a list of the images required.

I think rebuilt minikube to ensure that the images listed were all included in the list of images cached by minikube and on repeating the operation it still hung.

On further investigation is became clear that the list of images returned by "kubeadm config list images" was incorrect w.r.t the pause and coredns images:

The list command returns:

• coredns:1.1.3
• pause-amd64:3.1

The images actually used:

• coredns:1.2.2
• pause:3.1

When these images were added to the minikube binary a new minikube cluster started successfully.

What you expected to happen?

I expected kubeadm to return the images required to start the cluster.

How to reproduce it (as minimally and precisely as possible)?

Try to use "kubeadm init" to startup a new v1.11.0 k8s cluster while not connected to the internet having loaded the images returned by "kubeadm config list images --kubernetes-version=v1.11.0" and it will hang.

Startup a new (single node?) cluster using kubeadm while connected to the internet
Run "kubeadm config images list"
Note the identifiers for coredns and pause images.
Run "docker ps" and "docker images" on the node
Note the identifiers for coredns and pause images ebing used by the running containers.
They do not match.

Below are the results from inside the minkube VM

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/coredns 1.2.2 367cdc8433a4 2 months ago 39.2MB
....
k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 10 months ago 742kB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 10 months ago 742kB
.....

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
668db3d882a0 367cdc8433a4 "/coredns -conf /etc…" 57 seconds ago Up 55 seconds k8s_coredns_coredns-99b9bb8bd-ks74k_kube-system_66f0e87b-dc2e-11e8-aefb-080027f9ce4e_0
....
02ec7e7049e1 k8s.gcr.io/pause:3.1 "/pause" 59 seconds ago Up 57 seconds k8s_POD_storage-provisioner_kube-system_67b2c2d6-dc2e-11e8-aefb-080027f9ce4e_0
....

$ kubeadm config images list --kubernetes-version=v1.11.0
I1030 10:29:52.853477 6440 feature_gate.go:230] feature gates: &{map[]}
....
k8s.gcr.io/pause-amd64:3.1
...
k8s.gcr.io/coredns:1.1.3

Anything else we need to know?

Although this problem has been identified via minikube the issue is not specific to minikube and will be encountered whenever an attempt is made to run "kubeadm init" in a disconnected env using the image list returned from "kubeadm config images list"

/kind bug

[yagonobre]

This was fixed on kubeadm 1.12

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

$ kubeadm config images list --kubernetes-version=v1.11.0
k8s.gcr.io/kube-apiserver:v1.11.0
k8s.gcr.io/kube-controller-manager:v1.11.0
k8s.gcr.io/kube-scheduler:v1.11.0
k8s.gcr.io/kube-proxy:v1.11.0
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.18
k8s.gcr.io/coredns:1.2.2

[mlgibbons]

OK. That's good to know. Thanks for the info.

[neolit123]

yes, we also fixed a number of bugs.
please consider upgrading! thanks.