Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error enabling ingress addon #13249

Closed
matrixersp opened this issue Dec 28, 2021 · 11 comments
Closed

Error enabling ingress addon #13249

matrixersp opened this issue Dec 28, 2021 · 11 comments
Labels
addon/ingress kind/support Categorizes issue or PR as a support question.

Comments

@matrixersp
Copy link

What Happened?

I get the following error when I run minikube addons enable ingress:

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.22.3/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml: Process exited with status 1

Attach the log file

logs.txt

Operating System

Redhat/Fedora

Driver

Docker
@spowelljr
Copy link
Member

Looks like a container is not being created.

* * ==> kubelet <== * -- Logs begin at Mon 2021-12-27 14:11:51 UTC, end at Tue 2021-12-28 11:04:19 UTC. -- Dec 27 14:31:24 minikube kubelet[2177]: I1227 14:31:24.185788 2177 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="7021a68087b177e5a3dcd133770193249e8929c31309553056e05eb1ef8068c5" Dec 27 14:31:24 minikube kubelet[2177]: I1227 14:31:24.186185 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-patch--1-bs4jv through plugin: invalid network status for" Dec 27 14:31:24 minikube kubelet[2177]: I1227 14:31:24.187949 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-create--1-s4vkn through plugin: invalid network status for" Dec 27 14:31:24 minikube kubelet[2177]: I1227 14:31:24.188564 2177 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="2444ba91c2b4c92afd1abb36b04653d7cd6e6c27cab63be4376327c8410c58cb" Dec 27 14:31:24 minikube kubelet[2177]: E1227 14:31:24.878950 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:24 minikube kubelet[2177]: E1227 14:31:24.879147 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert podName:2c7ebaf6-6251-47ee-9ffc-22853e27145a nodeName:}" failed. No retries permitted until 2021-12-27 14:31:26.879099615 +0000 UTC m=+1153.183654432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert") pod "ingress-nginx-controller-54bfb9bb-7d5rq" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a") : secret "ingress-nginx-admission" not found Dec 27 14:31:25 minikube kubelet[2177]: I1227 14:31:25.205476 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-patch--1-bs4jv through plugin: invalid network status for" Dec 27 14:31:25 minikube kubelet[2177]: I1227 14:31:25.212160 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-create--1-s4vkn through plugin: invalid network status for" Dec 27 14:31:26 minikube kubelet[2177]: E1227 14:31:26.896398 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:26 minikube kubelet[2177]: E1227 14:31:26.896770 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert podName:2c7ebaf6-6251-47ee-9ffc-22853e27145a nodeName:}" failed. No retries permitted until 2021-12-27 14:31:30.896609569 +0000 UTC m=+1157.201164381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert") pod "ingress-nginx-controller-54bfb9bb-7d5rq" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a") : secret "ingress-nginx-admission" not found Dec 27 14:31:30 minikube kubelet[2177]: E1227 14:31:30.930562 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:30 minikube kubelet[2177]: E1227 14:31:30.930821 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert podName:2c7ebaf6-6251-47ee-9ffc-22853e27145a nodeName:}" failed. No retries permitted until 2021-12-27 14:31:38.930764982 +0000 UTC m=+1165.235319768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert") pod "ingress-nginx-controller-54bfb9bb-7d5rq" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a") : secret "ingress-nginx-admission" not found Dec 27 14:31:38 minikube kubelet[2177]: E1227 14:31:38.999204 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:39 minikube kubelet[2177]: E1227 14:31:38.999464 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert podName:2c7ebaf6-6251-47ee-9ffc-22853e27145a nodeName:}" failed. No retries permitted until 2021-12-27 14:31:54.999406701 +0000 UTC m=+1181.303961482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert") pod "ingress-nginx-controller-54bfb9bb-7d5rq" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a") : secret "ingress-nginx-admission" not found Dec 27 14:31:49 minikube kubelet[2177]: I1227 14:31:49.390884 2177 topology_manager.go:200] "Topology Admit Handler" Dec 27 14:31:49 minikube kubelet[2177]: I1227 14:31:49.492309 2177 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpc7\" (UniqueName: \"kubernetes.io/projected/70163996-8629-4fa3-b553-2402efeeeafd-kube-api-access-dkpc7\") pod \"ingress-nginx-controller-5f66978484-4jnjs\" (UID: \"70163996-8629-4fa3-b553-2402efeeeafd\") " Dec 27 14:31:49 minikube kubelet[2177]: I1227 14:31:49.492564 2177 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert\") pod \"ingress-nginx-controller-5f66978484-4jnjs\" (UID: \"70163996-8629-4fa3-b553-2402efeeeafd\") " Dec 27 14:31:49 minikube kubelet[2177]: E1227 14:31:49.594477 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:49 minikube kubelet[2177]: E1227 14:31:49.594716 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert podName:70163996-8629-4fa3-b553-2402efeeeafd nodeName:}" failed. No retries permitted until 2021-12-27 14:31:50.094627628 +0000 UTC m=+1176.399182464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert") pod "ingress-nginx-controller-5f66978484-4jnjs" (UID: "70163996-8629-4fa3-b553-2402efeeeafd") : secret "ingress-nginx-admission" not found Dec 27 14:31:50 minikube kubelet[2177]: E1227 14:31:50.098418 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:50 minikube kubelet[2177]: E1227 14:31:50.098606 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert podName:70163996-8629-4fa3-b553-2402efeeeafd nodeName:}" failed. No retries permitted until 2021-12-27 14:31:51.098554172 +0000 UTC m=+1177.403108963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert") pod "ingress-nginx-controller-5f66978484-4jnjs" (UID: "70163996-8629-4fa3-b553-2402efeeeafd") : secret "ingress-nginx-admission" not found Dec 27 14:31:51 minikube kubelet[2177]: E1227 14:31:51.107245 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:51 minikube kubelet[2177]: E1227 14:31:51.107534 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert podName:70163996-8629-4fa3-b553-2402efeeeafd nodeName:}" failed. No retries permitted until 2021-12-27 14:31:53.107450719 +0000 UTC m=+1179.412005555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert") pod "ingress-nginx-controller-5f66978484-4jnjs" (UID: "70163996-8629-4fa3-b553-2402efeeeafd") : secret "ingress-nginx-admission" not found Dec 27 14:31:53 minikube kubelet[2177]: E1227 14:31:53.124701 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:53 minikube kubelet[2177]: E1227 14:31:53.124968 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert podName:70163996-8629-4fa3-b553-2402efeeeafd nodeName:}" failed. No retries permitted until 2021-12-27 14:31:57.124916769 +0000 UTC m=+1183.429471546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert") pod "ingress-nginx-controller-5f66978484-4jnjs" (UID: "70163996-8629-4fa3-b553-2402efeeeafd") : secret "ingress-nginx-admission" not found Dec 27 14:31:55 minikube kubelet[2177]: E1227 14:31:55.042553 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:55 minikube kubelet[2177]: E1227 14:31:55.042833 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert podName:2c7ebaf6-6251-47ee-9ffc-22853e27145a nodeName:}" failed. No retries permitted until 2021-12-27 14:32:27.04275028 +0000 UTC m=+1213.347305084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert") pod "ingress-nginx-controller-54bfb9bb-7d5rq" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a") : secret "ingress-nginx-admission" not found Dec 27 14:31:57 minikube kubelet[2177]: E1227 14:31:57.162402 2177 secret.go:195] Couldn't get secret ingress-nginx/ingress-nginx-admission: secret "ingress-nginx-admission" not found Dec 27 14:31:57 minikube kubelet[2177]: E1227 14:31:57.162666 2177 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert podName:70163996-8629-4fa3-b553-2402efeeeafd nodeName:}" failed. No retries permitted until 2021-12-27 14:32:05.162573874 +0000 UTC m=+1191.467128679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70163996-8629-4fa3-b553-2402efeeeafd-webhook-cert") pod "ingress-nginx-controller-5f66978484-4jnjs" (UID: "70163996-8629-4fa3-b553-2402efeeeafd") : secret "ingress-nginx-admission" not found Dec 27 14:31:59 minikube kubelet[2177]: I1227 14:31:59.595289 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-create--1-s4vkn through plugin: invalid network status for" Dec 27 14:31:59 minikube kubelet[2177]: I1227 14:31:59.614323 2177 scope.go:110] "RemoveContainer" containerID="a837b5da2a9fce0842f9a5f13950d9ce7a5b4f04380cdf37184fedb4f327ac3f" Dec 27 14:32:00 minikube kubelet[2177]: I1227 14:32:00.637643 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-admission-patch--1-bs4jv through plugin: invalid network status for" Dec 27 14:32:00 minikube kubelet[2177]: I1227 14:32:00.647173 2177 scope.go:110] "RemoveContainer" containerID="017e9e185481be7264966482265d2a12ac95ae4b30bc500fd00e0033a294efa0" Dec 27 14:32:00 minikube kubelet[2177]: I1227 14:32:00.660102 2177 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="2444ba91c2b4c92afd1abb36b04653d7cd6e6c27cab63be4376327c8410c58cb" Dec 27 14:32:01 minikube kubelet[2177]: I1227 14:32:01.671224 2177 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="7021a68087b177e5a3dcd133770193249e8929c31309553056e05eb1ef8068c5" Dec 27 14:32:01 minikube kubelet[2177]: I1227 14:32:01.801769 2177 reconciler.go:196] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlscn\" (UniqueName: \"kubernetes.io/projected/0671cf29-876d-413e-84f3-cc44dca3e975-kube-api-access-nlscn\") pod \"0671cf29-876d-413e-84f3-cc44dca3e975\" (UID: \"0671cf29-876d-413e-84f3-cc44dca3e975\") " Dec 27 14:32:01 minikube kubelet[2177]: I1227 14:32:01.808450 2177 operation_generator.go:866] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0671cf29-876d-413e-84f3-cc44dca3e975-kube-api-access-nlscn" (OuterVolumeSpecName: "kube-api-access-nlscn") pod "0671cf29-876d-413e-84f3-cc44dca3e975" (UID: "0671cf29-876d-413e-84f3-cc44dca3e975"). InnerVolumeSpecName "kube-api-access-nlscn". PluginName "kubernetes.io/projected", VolumeGidValue "" Dec 27 14:32:01 minikube kubelet[2177]: I1227 14:32:01.902488 2177 reconciler.go:319] "Volume detached for volume \"kube-api-access-nlscn\" (UniqueName: \"kubernetes.io/projected/0671cf29-876d-413e-84f3-cc44dca3e975-kube-api-access-nlscn\") on node \"minikube\" DevicePath \"\"" Dec 27 14:32:02 minikube kubelet[2177]: I1227 14:32:02.811391 2177 reconciler.go:196] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clndw\" (UniqueName: \"kubernetes.io/projected/c9161ba8-ae31-4eb7-86f9-25cad85ae10a-kube-api-access-clndw\") pod \"c9161ba8-ae31-4eb7-86f9-25cad85ae10a\" (UID: \"c9161ba8-ae31-4eb7-86f9-25cad85ae10a\") " Dec 27 14:32:02 minikube kubelet[2177]: I1227 14:32:02.817980 2177 operation_generator.go:866] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9161ba8-ae31-4eb7-86f9-25cad85ae10a-kube-api-access-clndw" (OuterVolumeSpecName: "kube-api-access-clndw") pod "c9161ba8-ae31-4eb7-86f9-25cad85ae10a" (UID: "c9161ba8-ae31-4eb7-86f9-25cad85ae10a"). InnerVolumeSpecName "kube-api-access-clndw". PluginName "kubernetes.io/projected", VolumeGidValue "" Dec 27 14:32:02 minikube kubelet[2177]: I1227 14:32:02.912817 2177 reconciler.go:319] "Volume detached for volume \"kube-api-access-clndw\" (UniqueName: \"kubernetes.io/projected/c9161ba8-ae31-4eb7-86f9-25cad85ae10a-kube-api-access-clndw\") on node \"minikube\" DevicePath \"\"" Dec 27 14:32:05 minikube kubelet[2177]: I1227 14:32:05.862844 2177 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0856cd59daf7b19f1a23f9bea63386764182d581879e4d9afa3a7d6806253a05" Dec 27 14:32:05 minikube kubelet[2177]: I1227 14:32:05.863008 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-5f66978484-4jnjs through plugin: invalid network status for" Dec 27 14:32:06 minikube kubelet[2177]: I1227 14:32:06.880393 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-5f66978484-4jnjs through plugin: invalid network status for" Dec 27 14:32:27 minikube kubelet[2177]: I1227 14:32:27.949726 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-54bfb9bb-7d5rq through plugin: invalid network status for" Dec 27 14:32:28 minikube kubelet[2177]: I1227 14:32:28.163709 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-54bfb9bb-7d5rq through plugin: invalid network status for" Dec 27 14:36:55 minikube kubelet[2177]: I1227 14:36:55.660400 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-5f66978484-4jnjs through plugin: invalid network status for" Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.110222 2177 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for ingress-nginx/ingress-nginx-controller-54bfb9bb-7d5rq through plugin: invalid network status for" Dec 27 14:38:25 minikube kubelet[2177]: E1227 14:38:25.171214 2177 kuberuntime_container.go:589] "PreStop hook failed" err="command '/wait-shutdown' exited with 137: " pod="ingress-nginx/ingress-nginx-controller-54bfb9bb-7d5rq" podUID=2c7ebaf6-6251-47ee-9ffc-22853e27145a containerName="controller" containerID="docker://d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.669616 2177 reconciler.go:196] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8jzs\" (UniqueName: \"kubernetes.io/projected/2c7ebaf6-6251-47ee-9ffc-22853e27145a-kube-api-access-q8jzs\") pod \"2c7ebaf6-6251-47ee-9ffc-22853e27145a\" (UID: \"2c7ebaf6-6251-47ee-9ffc-22853e27145a\") " Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.669714 2177 reconciler.go:196] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert\") pod \"2c7ebaf6-6251-47ee-9ffc-22853e27145a\" (UID: \"2c7ebaf6-6251-47ee-9ffc-22853e27145a\") " Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.672715 2177 operation_generator.go:866] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2c7ebaf6-6251-47ee-9ffc-22853e27145a" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGidValue "" Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.672723 2177 operation_generator.go:866] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7ebaf6-6251-47ee-9ffc-22853e27145a-kube-api-access-q8jzs" (OuterVolumeSpecName: "kube-api-access-q8jzs") pod "2c7ebaf6-6251-47ee-9ffc-22853e27145a" (UID: "2c7ebaf6-6251-47ee-9ffc-22853e27145a"). InnerVolumeSpecName "kube-api-access-q8jzs". PluginName "kubernetes.io/projected", VolumeGidValue "" Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.770084 2177 reconciler.go:319] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c7ebaf6-6251-47ee-9ffc-22853e27145a-webhook-cert\") on node \"minikube\" DevicePath \"\"" Dec 27 14:38:25 minikube kubelet[2177]: I1227 14:38:25.770204 2177 reconciler.go:319] "Volume detached for volume \"kube-api-access-q8jzs\" (UniqueName: \"kubernetes.io/projected/2c7ebaf6-6251-47ee-9ffc-22853e27145a-kube-api-access-q8jzs\") on node \"minikube\" DevicePath \"\"" Dec 27 14:38:26 minikube kubelet[2177]: I1227 14:38:26.146778 2177 scope.go:110] "RemoveContainer" containerID="d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" Dec 27 14:38:26 minikube kubelet[2177]: I1227 14:38:26.162271 2177 scope.go:110] "RemoveContainer" containerID="d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" Dec 27 14:38:26 minikube kubelet[2177]: E1227 14:38:26.162991 2177 remote_runtime.go:334] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" containerID="d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" Dec 27 14:38:26 minikube kubelet[2177]: I1227 14:38:26.163049 2177 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:docker ID:d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee} err="failed to get container status \"d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee\": rpc error: code = Unknown desc = Error: No such container: d729941638cefd626d6f3d02d9dc72b28f486ce0784b5d95d6e096597aa976ee" Dec 27 14:38:28 minikube kubelet[2177]: I1227 14:38:28.165459 2177 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=2c7ebaf6-6251-47ee-9ffc-22853e27145a path="/var/lib/kubelet/pods/2c7ebaf6-6251-47ee-9ffc-22853e27145a/volumes"

Could you try running minikube addons enable ingress --alsologtostderr and upload the output of the command, thank you!

@spowelljr spowelljr added addon/ingress kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. labels Dec 28, 2021
@matrixersp
Copy link
Author

minikube-log.txt

@JavierPalaciosNexplore
Copy link

This is also happening to me on a github workflow. Is a rarely used one, and last execution on Nov 3 worked (hope minikube version can be traced by virtual-environments history. In any case, I attach the initial part of the workflow below to make easier to reproduce.

jobs:
  validate:
    runs-on: ubuntu-20.04
    steps:
      - name: Start minikube
        run: |
          rm -rf ~/.minikube ~/.kube/cache
          minikube config set kubernetes-version 1.19.0
          minikube start
          kubectl -n kube-system wait pod --all --timeout=180s --for=condition=Ready
          minikube addons enable ingress
          kubectl -n ingress-nginx wait pod -l app.kubernetes.io/component=controller --timeout=120s --for=condition=Ready

The rm and kube-system wait are new additions, naive (and failed) attempts to workaround this issue.

@JavierPalaciosNexplore
Copy link

It seems that #13088, #13127 and #13130 are duplicated of this one (I joined the newer one instead of running down the list)

@spowelljr
Copy link
Member

Thanks for the updates!

So the main issues seems to be:

"/etc/kubernetes/addons/ingress-deploy.yaml": Job.batch "ingress-nginx-admission-create" is invalid: spec.template: Invalid value: ... field is immutable

It seems we're trying to recreate/update an existing job, which the job needs to be deleted first.

We have a couple people actively working on improving the ingress addon:
#13439
#12904

I reached out to them about this issue and will update you when I have some details.

@spowelljr spowelljr removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Jan 26, 2022
@prezha
Copy link
Contributor

prezha commented Jan 26, 2022

@matrixersp @JavierPalaciosNexplore thank you for your responses so far!

could you please try if the changes in pr #13439 would resolve issues for you and share results?

i got it working with opensuse tumbleweed + minikube v1.25.1 + docker v20.10.12 + k8s v1.19.0:

❯ minikube start --kubernetes-version=v1.19.0 --wait all
😄  minikube v1.25.1 on Opensuse-Tumbleweed 
✨  Automatically selected the docker driver. Other choices: kvm2, virtualbox, ssh
💨  For improved Docker performance, enable the overlay Linux kernel module using 'modprobe overlay'
❗  docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
🐳  Preparing Kubernetes v1.19.0 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    > kubeadm.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubeadm: 37.30 MiB / 37.30 MiB [-------------] 100.00% 10.39 MiB p/s 3.8s
    > kubectl: 41.01 MiB / 41.01 MiB [-------------] 100.00% 12.79 MiB p/s 3.4s
    > kubelet: 104.88 MiB / 104.88 MiB [-----------] 100.00% 13.44 MiB p/s 8.0s
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
E0126 21:34:25.306617   21191 pod_ready.go:66] WaitExtra: waitPodCondition: error getting pod "coredns-f9fd979d6-2lr5p" in "kube-system" namespace (skipping!): pods "coredns-f9fd979d6-2lr5p" not found

❗  /home/prezha/bin/k8s/kubectl is version 1.23.3, which may have incompatibilites with Kubernetes 1.19.0.
    ▪ Want kubectl v1.19.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

note1: error about coredns deployment above can be ignored (being replaced with the other having replica set of 1 instead of the default of 2...)
note2: @JavierPalaciosNexplore you can use minikube start --kubernetes-version=v1.19.0 --wait all instead of the kubectl -n kube-system wait pod --all --timeout=180s --for=condition=Ready

❯ minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

❯ kubectl get pods -o wide -A
NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
ingress-nginx   ingress-nginx-admission-create-ts77s       0/1     Completed   0          36s     172.17.0.4     minikube   <none>           <none>
ingress-nginx   ingress-nginx-admission-patch-4xh65        0/1     Completed   0          35s     172.17.0.2     minikube   <none>           <none>
ingress-nginx   ingress-nginx-controller-f975c46f6-d27jd   1/1     Running     0          36s     172.17.0.2     minikube   <none>           <none>
kube-system     coredns-f9fd979d6-49wqv                    1/1     Running     0          2m26s   172.17.0.3     minikube   <none>           <none>
kube-system     etcd-minikube                              1/1     Running     0          2m34s   192.168.49.2   minikube   <none>           <none>
kube-system     kube-apiserver-minikube                    1/1     Running     0          2m34s   192.168.49.2   minikube   <none>           <none>
kube-system     kube-controller-manager-minikube           1/1     Running     0          2m34s   192.168.49.2   minikube   <none>           <none>
kube-system     kube-proxy-4jx9h                           1/1     Running     0          2m26s   192.168.49.2   minikube   <none>           <none>
kube-system     kube-scheduler-minikube                    1/1     Running     0          2m34s   192.168.49.2   minikube   <none>           <none>
kube-system     storage-provisioner                        1/1     Running     0          2m25s   192.168.49.2   minikube   <none>           <none>

@prezha prezha mentioned this issue Jan 26, 2022
Closed
@JavierPalaciosNexplore
Copy link

Should I build my own minikube locally, or there is any way to test the changes on that pull-reques?

And thanks for the shortened startup command @prezha

@spowelljr
Copy link
Member

@JavierPalaciosNexplore You can checkout his PR and build it yourself locally or you can use the binary that was built during testing for his PR.

https://storage.googleapis.com/minikube-builds/13439/minikube-linux-amd64

@JavierPalaciosNexplore
Copy link

I've made a couple of tests for now with that executable, and no issue with ingress for now.

@spowelljr
Copy link
Member

Glad to hear!

@prezha
Copy link
Contributor

prezha commented Feb 7, 2022

@JavierPalaciosNexplore thank you for taking the time to test and report results back - great that it worked for you!

as #13439 was merged, i'll close this issue now

/close

@prezha prezha closed this as completed Feb 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
addon/ingress kind/support Categorizes issue or PR as a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@prezha @matrixersp @spowelljr @JavierPalaciosNexplore