controlPlaneEndpoint=localhost breaks clusterapi-apiserver: webhook.go connection refused

[joejulian]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment: x86_64 linux

Minikube version (use minikube version): >= v0.28.1

• OS (e.g. from /etc/os-release): ubuntu, varying versions
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): kvm2
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): >= v0.28.1

What happened:
cluster-api errors with the following

ERROR: logging before flag.Parse: E0814 22:20:52.810904       1 webhook.go:192] Failed to make webhook authorizer request: Post https://localhost:8443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: dial tcp 127.0.0.1:8443: connect: connection refused
ERROR: logging before flag.Parse: E0814 22:20:52.811051       1 errors.go:90] Post https://localhost:8443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: dial tcp 127.0.0.1:8443: connect: connection refused

What you expected to happen:
With 0.28.0 it connected to the apiserver as expected.

How to reproduce it (as minimally and precisely as possible):
Start minikube and install the clusterapi controller:
curl -L https://git.io/clusterapi | kubectl apply -f -

Output of minikube logs (if applicable):

Anything else do we need to know:
This behavior changed with the merge of this line. Removing the line reverts the behavior and allows the application to function correctly.

https://github.com/kubernetes/minikube/pull/2943/files#diff-791cd4f1bc5624fa1aec26b76983b601R32

[tstromberg]

Hey @joejulian - I'd like to help push this forward. I don't yet know the motivation behind the original change to set controlPlaneEndpoint: localhost, but if minikube works without it, I'd happily make or review a PR that reverts this line.

I also don't know much about the cluster-api or how to get it up and running. I tried the repro instructions:

% curl -L https://git.io/clusterapi | kubectl apply -f -
statefulset.apps/etcd-clusterapi unchanged
deployment.extensions/clusterapi-apiserver unchanged

But ended with a pod stuck in ContainerCreating. Is this a symptom of what you are seeing? If so, how do I see the log with the errors you mentioned?

NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE
default       clusterapi-apiserver-7787d8f7cf-5lmvk   0/1       MatchNodeSelector   0          12m
default       clusterapi-apiserver-7787d8f7cf-f86c7   0/1       ContainerCreating   0          10m

Events:
  Type     Reason                 Age               From               Message
  ----     ------                 ----              ----               -------
  Normal   Scheduled              10m               default-scheduler  Successfully assigned clusterapi-apiserver-7787d8f7cf-5lmvk to minikube
  Normal   SuccessfulMountVolume  10m               kubelet, minikube  MountVolume.SetUp succeeded for volume "config"
  Normal   SuccessfulMountVolume  10m               kubelet, minikube  MountVolume.SetUp succeeded for volume "certs"
  Normal   SuccessfulMountVolume  10m               kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-6xz4g"
  Warning  FailedMount            9m (x5 over 10m)  kubelet, minikube  MountVolume.SetUp failed for volume "cluster-apiserver-certs" : secrets "cluster-apiserver-certs" not found
  Warning  MatchNodeSelector      7m                kubelet, minikube  Predicate MatchNodeSelector failed

[joejulian]

Yes, it is.

Look at https://github.com/samsung-cnct/cluster-api-provider-ssh which is the implementation I was using and has clear instructions.

The pods will start, but when you look at the logs, they will fail to connect to the api server because they're trying to connect to localhost:6443 which, of course, is not going to connect from inside a pod.

[joejulian]

Reading back, I see I had it fail with the vanilla cluster-api. It's been a while and I'd forgotten.

Looks like it wants a secret volume that it didn't need back when I wrote that.

[tstromberg]

Hey Joe - is this still an issue?

If so, can you help with the reproduction instructions? I ran curl -L https://git.io/clusterapi | kubectl apply -f - but it just resulted in a pod that was missing the secrets volume:

  Normal   Scheduled    2m                default-scheduler  Successfully assigned default/clusterapi-apiserver-5765fc9bb9-946qr to minikube
  Warning  FailedMount  44s               kubelet, minikube  Unable to mount volumes for pod "clusterapi-apiserver-5765fc9bb9-946qr_default(a454c3f6-1f67-11e9-a369-a41d417078d8)": timeout expired waiting for volumes to attach or mount for pod "default"/"clusterapi-apiserver-5765fc9bb9-946qr". list of unmounted volumes=[cluster-apiserver-certs]. list of unattached volumes=[cluster-apiserver-certs config certs default-token-wc9mb]
  Warning  FailedMount  39s (x9 over 2m)  kubelet, minikube  MountVolume.SetUp failed for volume "cluster-apiserver-certs" : secret "cluster-apiserver-certs" not found

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

I'm closing this issue as it hasn't seen activity in awhile, and it's unclear if this issue still exists. If this issue does continue to exist in the most recent release of minikube, please feel free to re-open it.

Thank you for opening the issue!

[vikaschoudhary16]

/reopen

I am facing this issue because of this line i.e setting controlPlaneEndpoint to localhost. I am running like this:

 minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd --kubernetes-version v1.12.0 --v 5  --alsologtostderr 

minikube version is 1.1.0

Error which i get is this:

 kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong API Server URL

This error is from kubeadm. kubeadm compares server url from the kubeadm.yaml and from the kubeconfig file on the disk. They differ like this:

 From disk kubeconfig:  https://192.168.1.103:8443
From the parameters in kubeadm.yaml:  https://localhost:8443

[k8s-ci-robot]

@vikaschoudhary16: Reopened this issue.

In response to this:

/reopen

I am facing this issue because of this line i.e setting controlPlaneEndpoint to localhost. I am running like this:

minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd --kubernetes-version v1.12.0 --v 5  --alsologtostderr 

minikube version is 1.1.0

Error which i get is this:

kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong API Server URL

This error is from kubeadm. kubeadm compares server url from the kubeadm.yaml and from the kubeconfig file on the disk. They differ like this:

From disk kubeconfig:  https://192.168.1.103:8443
From the parameters in kubeadm.yaml:  https://localhost:8443

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[joejulian]

Since I'm no longer at Samsung and cluster-api-provider-ssh is abandoned and all the cluster-api development uses kind now, I'm fine with closing this.

[tstromberg]

As a test, I removed this line:

minikube/pkg/minikube/bootstrapper/kubeadm/templates.go

Line 133 in 01f399b

controlPlaneEndpoint: localhost:{{.APIServerPort}}

When I tried to restart an existing v1.16 cluster, kubeadm emitted the error:

 [kubeconfig] Using kubeconfig folder "/etc/kubernetes"

stderr: error execution phase kubeconfig/admin: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong API Server URL
To see the stack trace of this error execute with --v=5 or higher

On a clean cluster it appears to work fine. I'm OK with removing this line, but some work will have to be done to cleanly handle existing clusters.

[medyagh]

I believe @vikaschoudhary16 your issue has a workaround

#4172

[medyagh]

I believe this issue was fixed this PR by #5427