Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error During Ingress Add-on Enablement #12501

Closed
sattsb opened this issue Sep 17, 2021 · 10 comments
Closed

Error During Ingress Add-on Enablement #12501

sattsb opened this issue Sep 17, 2021 · 10 comments
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@sattsb
Copy link

sattsb commented Sep 17, 2021 •
edited
Loading

Steps to reproduce the issue:

  1. Execute minikube addons enable ingress

Errors out with the following message

❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.22.1/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml: Process exited with status 1
stdout:

stderr:
Unable to connect to the server: net/http: TLS handshake timeout
 waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
[minikube_addons_e74c939bb4977c35a735826431278ceb07d07a67_0.log](https://github.com/kubernetes/minikube/files/7187355/minikube_addons_e74c939bb4977c35a735826431278ceb07d07a67_0.log)
@spowelljr
Copy link
Member

Hi @sattsb, thanks for reporting your issue with minikube!

Based on your logs if looks like you're using minikube version v1.23.0 which has an ingress regression. Please try updating to v1.23.1 and trying again and see if that solves your issue.

@spowelljr spowelljr added addon/ingress kind/support Categorizes issue or PR as a support question. labels Sep 17, 2021
@ChenLin12138
Copy link

ChenLin12138 commented Sep 30, 2021 •
edited
Loading

@spowelljr
I have same issue even I used minikubev1.23.2.

C:\Program Files\Kubernetes\Minikube>minikube start
* minikube v1.23.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
* Kubernetes 1.22.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.22.2
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Restarting existing docker container for "minikube" ...
! This container is having trouble accessing https://k8s.gcr.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.22.1 on Docker 20.10.8 ...
* Verifying Kubernetes components...
! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 2.5992634s
* Restarting the docker service may improve performance.
  - Using image kubernetesui/metrics-scraper:v1.0.7
  - Using image kubernetesui/dashboard:v2.3.1
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, dashboard, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
C:\Program Files\Kubernetes\Minikube>minikube addons enable ingress
* After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
  - Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
* Verifying ingress addon...
X Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
*
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                │
│    * If the above advice does not help, please let us know:                                                    │
│      https://github.com/kubernetes/minikube/issues/new/choose                                                  │
│                                                                                                                │
│    * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                       │
│    * Please also attach the following file to the GitHub issue:                                                │
│    * - C:\Users\chenli2\AppData\Local\Temp\2\minikube_addons_e0e7a0581240141c35e4e8309b248d753611d39a_0.log    │
│                                                                                                                │

@spowelljr
Copy link
Member

Hi @ChenLin12138, the issue you're experiencing is a different know issue (#10544).

The current work around for it is to run minikube delete --all --purge.

@ChenLin12138
Copy link

@spowelljr Thanks you so much, I have try your solutions but it still not work. Any other advise?
delete action:

C:\Program Files\Kubernetes\Minikube>minikube delete --all --purge
* Deleting "minikube" in docker ...
* Removing C:\Users\chenli2\.minikube\machines\minikube ...
* Removed all traces of the "minikube" cluster.
* Successfully deleted all profiles
* Successfully purged minikube directory located at - [C:\Users\chenli2\.minikube]

start action:

C:\Program Files\Kubernetes\Minikube>minikube start
* minikube v1.23.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
* Automatically selected the docker driver
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.22.2 preload ...
    > preloaded-images-k8s-v13-v1...: 511.84 MiB / 511.84 MiB  100.00% 13.22 Mi
    > index.docker.io/kicbase/sta...: 355.40 MiB / 355.40 MiB  100.00% 8.03 MiB
! minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.27, but successfully downloaded docker.io/kicbase/stable:v0.0.27 as a fallback image
* Creating docker container (CPUs=2, Memory=1987MB) ...
! This container is having trouble accessing https://k8s.gcr.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.22.2 on Docker 20.10.8 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Verifying Kubernetes components...
! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 2.4368825s
* Restarting the docker service may improve performance.
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

enable ingress

C:\Program Files\Kubernetes\Minikube>minikube addons enable ingress
* After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
  - Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
* Verifying ingress addon...

X Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
*
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                │
│    * If the above advice does not help, please let us know:                                                    │
│      https://github.com/kubernetes/minikube/issues/new/choose                                                  │
│                                                                                                                │
│    * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                       │
│    * Please also attach the following file to the GitHub issue:                                                │
│    * - C:\Users\chenli2\AppData\Local\Temp\2\minikube_addons_e0e7a0581240141c35e4e8309b248d753611d39a_0.log    │
│                                                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

@sattsb
Copy link
Author

sattsb commented Oct 1, 2021

@spowelljr and @ChenLin12138 . what helped me was downgrading kubernetes so 1.19.X

@spowelljr
Copy link
Member

@ChenLin12138 I'm curious if docker is caching an image that's outdated and that's causing the issue, I'm not too sure if it will work but at least we can then rule it out.

After running minikube delete --all --purge try running docker system prune -a --volumes and then try starting and enabling ingress again if you don't mind, thanks!

@ChenLin12138
Copy link

@spowelljr
Thanks for your help, I have try it but still not work.
My os is Windows 10 Enterprise 20H2
Docker version Docker version 20.10.7, build f0df350
minikube v1.23.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042

C:\NotBackedUp\learn\k8s-learning-master\C5>minikube delete --all --purge
* Removed all traces of the "minikube" cluster.
* Successfully deleted all profiles
* Successfully purged minikube directory located at - [C:\Users\chenli2\.minikube]

C:\NotBackedUp\learn\k8s-learning-master\C5>docker system prune -a --volumes
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all volumes not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Containers:
4da577bab4ea198222eb37783f2a28025a9ba4a643e7fc206d9ce362e2376ad2
9748fd68ec1a68a06a5411145b1b69a9d96d03dc3c6d7c5368b2c468ace2904d

Deleted Networks:
minikube

Deleted Images:
untagged: kicbase/stable:v0.0.27
untagged: kicbase/stable@sha256:89b4738ee74ba28684676e176752277f0db46f57d27f0e08c3feec89311e22de
deleted: sha256:9fa1cc16ad6d525b8319e86c05a7fb228f97aeaf4bcda0033a709f018a2e3ee

C:\NotBackedUp\learn\k8s-learning-master\C5>minikube start
* minikube v1.23.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
* Automatically selected the docker driver
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.22.2 preload ...
    > preloaded-images-k8s-v13-v1...: 511.84 MiB / 511.84 MiB  100.00% 9.47 MiB
    > index.docker.io/kicbase/sta...: 355.40 MiB / 355.40 MiB  100.00% 8.76 MiB
! minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.27, but successfully downloaded docker.io/kicbase/stable:v0.0.27 as a fallback image
* Creating docker container (CPUs=2, Memory=1987MB) ...
! Executing "docker ps -a --format {{.Names}}" took an unusually long time: 3.8708489s
* Restarting the docker service may improve performance.
! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 3.8711615s
* Restarting the docker service may improve performance.
! This container is having trouble accessing https://k8s.gcr.io
* Preparing Kubernetes v1.22.2 on Docker 20.10.8 ...* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

C:\NotBackedUp\learn\k8s-learning-master\C5>minikube addons enable ingress
! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 4.0634015s
* Restarting the docker service may improve performance.
* After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
  - Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
  - Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
* Verifying ingress addon...

X Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
*
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                │
│    * If the above advice does not help, please let us know:                                                    │
│      https://github.com/kubernetes/minikube/issues/new/choose                                                  │
│                                                                                                                │
│    * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                       │
│    * Please also attach the following file to the GitHub issue:                                                │
│    * - C:\Users\chenli2\AppData\Local\Temp\1\minikube_addons_e0e7a0581240141c35e4e8309b248d753611d39a_0.log    │
│                                                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

@sattsb Let me downgrade minikube to 1.19.X and try it, thank you.

@spowelljr
Copy link
Member

Hi @ChenLin12138, could you run minikube addons enable ingress --alsologtostderr and include the logs, there will be more verbose logging.

@spowelljr spowelljr added the triage/needs-information Indicates an issue needs more information in order to work on it. label Oct 13, 2021
@prezha prezha mentioned this issue Oct 27, 2021
Merged
@sharifelgamal
Copy link
Collaborator

Yeah unfortunately all these logs tell us is that the ingress-nginx image used for the ingress addon didn't start up, but without extra logs, there's no real way for us to know why.

@sharifelgamal sharifelgamal added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Nov 3, 2021
@sharifelgamal
Copy link
Collaborator

Hey @ChenLin12138, I would try minikube v1.24.0, we fixed some more ingress issues here as well. If the issues continue, please paste the output of minikube addons enable ingress --alsologtostderr in a new issue.

@sattsb, please feel free to reopen this issue if your issues persist as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sattsb @ChenLin12138 @sharifelgamal @spowelljr