kubeadm.skip-phases=addon/kube-proxy is not honored when restarting an existing cluster

[twz123]

Steps to reproduce the issue:

#!/usr/bin/env sh

set -eux

minikube_start() {
  minikube start \
    --driver kvm \
    --extra-config=kubeadm.skip-phases=addon/kube-proxy
  #--kubernetes-version=v1.18.14 \
}

export LANG=C
export MINIKUBE_PROFILE=proof

minikube version
minikube delete || true
minikube_start

sleep 5
kubectl --context "$MINIKUBE_PROFILE" -n kube-system get daemonset -owide -l k8s-app=kube-proxy
sleep 5
kubectl --context "$MINIKUBE_PROFILE" -n kube-system get daemonset -owide -l k8s-app=kube-proxy

minikube stop
minikube_start

sleep 5
kubectl --context "$MINIKUBE_PROFILE" -n kube-system get daemonset -owide -l k8s-app=kube-proxy

Output is as follows for me:

+ export LANG=C
+ LANG=C
+ export MINIKUBE_PROFILE=proof
+ MINIKUBE_PROFILE=proof
+ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e
+ minikube delete
* Deleting "proof" in kvm2 ...
* Removed all traces of the "proof" cluster.
+ minikube_start
+ minikube start --driver kvm --extra-config=kubeadm.skip-phases=addon/kube-proxy
* [proof] minikube v1.22.0 on Nixos 21.05.20210831.60712d4 (Okapi)
  - MINIKUBE_PROFILE=proof
* Using the kvm2 driver based on user configuration
* Starting control plane node proof in cluster proof
* Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
  - kubeadm.skip-phases=addon/kube-proxy
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass

! /home/t.wieczorek/.local/bin/kubectl is version 1.19.14, which may have incompatibilites with Kubernetes 1.21.2.
  - Want kubectl v1.21.2? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "proof" cluster and "default" namespace by default
+ sleep 5
+ kubectl --context proof -n kube-system get daemonset -owide -l k8s-app=kube-proxy
No resources found in kube-system namespace.
+ sleep 5
+ kubectl --context proof -n kube-system get daemonset -owide -l k8s-app=kube-proxy
No resources found in kube-system namespace.
+ minikube stop
* Stopping node "proof"  ...
* 1 nodes stopped.
+ minikube_start
+ minikube start --driver kvm --extra-config=kubeadm.skip-phases=addon/kube-proxy
* [proof] minikube v1.22.0 on Nixos 21.05.20210831.60712d4 (Okapi)
  - MINIKUBE_PROFILE=proof
* Using the kvm2 driver based on existing profile
* Starting control plane node proof in cluster proof
* Restarting existing kvm2 VM for "proof" ...
* Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
  - kubeadm.skip-phases=addon/kube-proxy
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass

! /home/t.wieczorek/.local/bin/kubectl is version 1.19.14, which may have incompatibilites with Kubernetes 1.21.2.
  - Want kubectl v1.21.2? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "proof" cluster and "default" namespace by default
+ sleep 5
+ kubectl --context proof -n kube-system get daemonset -owide -l k8s-app=kube-proxy
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                          SELECTOR
kube-proxy   0         0         0       0            0           kubernetes.io/os=linux   5s    kube-proxy   k8s.gcr.io/kube-proxy:v1.21.2   k8s-app=kube-proxy

After the restart, the kube-proxy daemonset is installed, which shouldn't be the case.

minikube logs --file=minikube.log
minikube.log

[RA489]

/kind support

[spowelljr]

Hi @twz123, thanks for reporting your issue with minikube!

I was able to reproduce using your steps and can confirm this is a bug with minikube.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[spowelljr]

So the main issue is we do a different path on restart, and in that path is:

_, err := k.c.RunCmd(exec.Command("/bin/bash", "-c", fmt.Sprintf("%s phase addon all --config %s", baseCmd, conf)))

So we're hard coding installing all addons

[twz123]

@spowelljr Thnaks for fixing. Unfortunately, as a consequence of recreating the cluster, it's empty afterwards and all of its contents have to be recreated. This means that restarting such a cluster is not possible at all. Any chance to make restarts work again?

[spowelljr]

Once the revert takes place this can be revisited.