Kubeadm has incorrect images list to bootstrap k8s 1.11 without internet

[stepin]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: hardware
• OS (e.g. from /etc/os-release):
  VERSION="7.3 (Maipo)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="7.3"
  PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
  HOME_URL="https://www.redhat.com/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"

• Kernel (e.g. uname -a):
  Linux hostname 3.10.0-514.32.3.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Thu Sep 21 04:23:20 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

I'm installing Kubernetes cluster on bare-metal hw without direct access to internet.

Local images list from kubeadm:

# kubeadm config images list --kubernetes-version v1.11.0 --feature-gates CoreDNS=false
I0720 18:47:20.125324    1089 feature_gate.go:230] feature gates: &{map[]}
k8s.gcr.io/kube-apiserver-amd64:v1.11.0
k8s.gcr.io/kube-controller-manager-amd64:v1.11.0
k8s.gcr.io/kube-scheduler-amd64:v1.11.0
k8s.gcr.io/kube-proxy-amd64:v1.11.0
k8s.gcr.io/pause-amd64:3.1
k8s.gcr.io/etcd-amd64:3.2.18
k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.10

With this images kubeadm init --kubernetes-version v1.11.0 --feature-gates CoreDNS=false command fails with error:

I0720 19:09:18.950778    1816 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
...
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

Unfortunately, an error has occurred:
  timed out waiting for the condition

This error is likely caused by:
  - The kubelet is not running
  - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
  - No internet connection is available so the kubelet cannot pull or find the following control plane images:
    - k8s.gcr.io/kube-apiserver-amd64:v1.11.0
    - k8s.gcr.io/kube-controller-manager-amd64:v1.11.0
    - k8s.gcr.io/kube-scheduler-amd64:v1.11.0
    - k8s.gcr.io/etcd-amd64:3.2.18
    - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
      are downloaded locally and cached.

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
  - 'systemctl status kubelet'
  - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
  - 'docker ps -a | grep kube | grep -v pause'
  Once you have found the failing container, you can inspect its logs with:
  - 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

What you expected to happen?

K8s cluster initialised correctly.

How to reproduce it (as minimally and precisely as possible)?

1. Use machines without internet
2. Load docker images from list above
3. Install kubernetes rpm packages
4. Run kubeadm init command from above

Anything else we need to know?

To fix issue add 3 extra images to k8s nodes:

1. add pause image without amd suffix
2. I'm also added 2 more images for kubedns (I remember that it was required for previous versions and these images exists for current version of kubedns). Not tested without these images on k8s 1.11.

List of extra images to init cluster without internet:

k8s.gcr.io/pause:3.1
k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10

[luxas]

Indeed,

k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10

should be added. The pause image has been fixed in a patch release.
/assign @rosti @chuckha

[k8s-ci-robot]

@luxas: GitHub didn't allow me to assign the following users: rosti.

Note that only kubernetes members and repo collaborators can be assigned.
For more information please see the contributor guide

In response to this:

Indeed,

k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10

should be added. The pause image has been fixed in a patch release.
/assign @rosti @chuckha

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[timothysc]

/assign @liztio @timothysc

@chuckha is on a different adventure for a while in cluster-api.

[luxas]

@timothysc @rosti has a PR for this already which fixes this 👍 kubernetes/kubernetes#66499