featureGates: StoreCertsInSecrets fails in Kubeadm v1.11

[fabriziopandini]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

v1.11 (.debs, kubeadm binary e docker images build from master)

What happened?

kubeadm init with SelfHosting: true and featureGates:StoreCertsInSecrets: true fails with:

...
[self-hosted] creating self-hosted control plane
[self-hosted] Created TLS secret "ca" from ca.crt and ca.key
[self-hosted] Created TLS secret "apiserver" from apiserver.crt and apiserver.key
[self-hosted] Created TLS secret "apiserver-kubelet-client" from apiserver-kubelet-client.crt and apiserver-kubelet-client.key
[self-hosted] Created TLS secret "sa" from sa.pub and sa.key
[self-hosted] Created TLS secret "front-proxy-ca" from front-proxy-ca.crt and front-proxy-ca.key
[self-hosted] Created TLS secret "front-proxy-client" from front-proxy-client.crt and front-proxy-client.key
[self-hosted] Created secret for kubeconfig file "scheduler.conf"
[self-hosted] Created secret for kubeconfig file "controller-manager.conf"
[apiclient] Found 0 Pods for label selector k8s-app=self-hosted-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=self-hosted-kube-apiserver
[apiclient] The old Pod "kube-apiserver-kubeadm-test-master" is now removed (which is desired)
[apiclient] All control plane components are healthy after 0.002673 seconds
[self-hosted] self-hosted kube-apiserver ready after 4.715637 seconds
error creating self hosted control plane: timed out waiting for the condition

or

...
[self-hosted] self-hosted kube-apiserver ready after 6.712564 seconds
[apiclient] Found 0 Pods for label selector k8s-app=self-hosted-kube-controller-manager
[apiclient] Error getting Pods with label selector "k8s-app=self-hosted-kube-controller-manager" [Get https://10.10.10.11:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dself-hosted-kube-controller-manager: dial tcp 10.10.10.11:6443: connect: connection refused]

NB. kubeadm init with featureGates:SelfHosting: true only works properly, so I assume the issue should be related to featureGates:StoreCertsInSecrets: true

What you expected to happen?

Kubeadm init complete, control plane self-hosted and certs stored in secrets

How to reproduce it (as minimally and precisely as possible)?

use this config file:

---
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: "1.11.0"
api:
  advertiseAddress: "10.10.10.11"

bootstrapTokens:
- token: "abcdef.0123456789abcdef"
nodeRegistration:
  kubeletExtraArgs:
    node-ip: "10.10.10.11"
featureGates:
  SelfHosting: true
  StoreCertsInSecrets: true

[fabriziopandini]

See kubernetes/kubernetes#61322

[timothysc]

/assign @liztio

[tommyknows]

Can confirm this issue, it looks like the APIServer Container dies.
Docker logs:

Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0704 10:55:59.462879       1 server.go:703] external host was not specified, using 172.17.0.52
I0704 10:55:59.463400       1 server.go:145] Version: v1.11.0
I0704 10:56:01.733675       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0704 10:56:01.734483       1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0704 10:56:01.736564       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0704 10:56:01.736843       1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
F0704 10:56:01.741833       1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry [https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true false 1000 0xc42045e500 <nil> 5m0s 1m0s}), err (open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory)

It seems like the bootstrapping process does not create the secret:

[...]
[self-hosted] creating self-hosted control plane
[self-hosted] Created TLS secret "ca" from ca.crt and ca.key
[self-hosted] Created TLS secret "apiserver" from apiserver.crt and apiserver.key
[self-hosted] Created TLS secret "apiserver-kubelet-client" from apiserver-kubelet-client.crt and apiserver-kubelet-client.key
[self-hosted] Created TLS secret "sa" from sa.pub and sa.key
[self-hosted] Created TLS secret "front-proxy-ca" from front-proxy-ca.crt and front-proxy-ca.key
[self-hosted] Created TLS secret "front-proxy-client" from front-proxy-client.crt and front-proxy-client.key
[self-hosted] Created secret for kubeconfig file "scheduler.conf"
[self-hosted] Created secret for kubeconfig file "controller-manager.conf"
[..]

(these are all self-hosted messages)

[fabriziopandini]

@timothysc there is already this PR from @stealthybox trying to address this kubernetes/kubernetes#61323
/assign @stealthybox

Ps. Since we are going to deprecate selfhosting, also store certs in secrets will be deprecated too

[stealthybox]

ack -- commented that update on the PR

not going to work on this if it isn't going to be supported in the near future

[runiq]

Since we are going to deprecate selfhosting, also store certs in secrets will be deprecated too

@fabriziopandini Is there a link where I can learn more about this? This is the first time I've heard about it.

[timothysc]

Marked as deprecated in 1.12