Exposing endpoint with "minikube service <svc>" does not work

[anowak-ct]

What Happened?

I am new to k8s and minikube. I followed the install instructions for windows,. Unfortunately, I cannot get the "echoserver" test to work as outlined in https://minikube.sigs.k8s.io/docs/handbook/controls/

I am using cygwin in Windows 10 (latest updates installed) with alias kubectl='minikube kubectl --'
Docker is using WSL2

Steps and output:

• minikube start

* minikube v1.25.2 on Microsoft Windows 10 Pro 10.0.19043 Build 19043
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Updating the running docker "minikube" container ...
* Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
  - kubelet.housekeeping-interval=5m
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

• minikube dashboard

* Enabling dashboard ...
  - Using image kubernetesui/dashboard:v2.3.1
  - Using image kubernetesui/metrics-scraper:v1.0.7
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
* Opening http://127.0.0.1:63164/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...

--> This worked fine. I can use the K8s UI in the browser

• kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4

deployment.apps/hello-minikube created

• kubectl expose deployment hello-minikube --type=NodePort --port=8080

service/hello-minikube exposed

• minikube service hello-minikube

* Starting tunnel for service hello-minikube.
* Opening service default/hello-minikube in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it.
* Stopping tunnel for service hello-minikube.

--> This opened my browser with url http://192.168.49.2:32473/ but it could not connect (connection timeout). I did not close the terminal and the command was still running.

• kubectl get service -o wide (The command minikube service hello-minikube was still running)

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE     SELECTOR
hello-minikube   NodePort    10.108.210.90   <none>        8080:32473/TCP   28m     app=hello-minikube
kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP          3h17m   <none>

• minikube logs --file=log.txt

--> See log attached

• netstat -na | grep 32473
  --> no output. Hence, port was not occupied.

Observations

The minikube.log is showing lots of errors or failures. I cannot tell which are benign which are real errors. However, I noticed messages which do sound like errors to me:

==> kubelet <==

 1968 docker_sandbox.go:402] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for default/hello-minikube-7bc9d7884c-cmznw through plugin: invalid network status for"

==> kube-proxy <==

E0504 16:31:05.726182       1 proxier.go:1600] "can't open port, skipping it" err="listen tcp4 :32473: bind: address already in use" port={Description:nodePort for default/hello-minikube IP: IPFamily:4 Port:32473 Protocol:TCP}

Running minikube tunnel did not work either.

I read in some blogs that there might be problems with Docker and WSL2....? I am using Docker alone in other projects with no problems whatsoever.

Any help is greatly appreciated.

Attach the log file

log.txt

Operating System

No response

Driver

No response

[anowak-ct]

Update:
The alternative to minikube service <svc> seems to work! I.e. using kubectl to forward the port:

kubectl port-forward service/hello-minikube 7080:8080

Secondly, doing a loadbalancer deployment as described in https://minikube.sigs.k8s.io/docs/start/ works as well:

• kubectl create deployment balanced --image=k8s.gcr.io/echoserver:1.4

deployment.apps/balanced created

• kubectl expose deployment balanced --type=LoadBalancer --port=8080

service/balanced exposed

• minikube.exe tunnel

* Tunnel successfully started

* NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...

* Starting tunnel for service balanced.
* Stopping tunnel for service balanced.

--> Opening http://localhost:8080/ in browser showed expected output.

So, seems like problem is just related to minikube service <svc>?

[zhan9san]

hi @anowak-ct

Could you help verify whether #13806 address this issue？

[klaases]

Hi @anowak-ct, did you have a chance to see if #13806 helps resolve this issue?

[BradKnowles]

I tested the new documentation, but it did not seem to resolve the issue for me.

minikube start --embed-certs

😄  minikube v1.25.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4
kubectl expose deployment hello-minikube --type=NodePort --port=8080

I ran minikube dashboard to verify everything was green.

kubectl get svc

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-minikube   NodePort    10.97.188.166   <none>        8080:30462/TCP   3m1s
kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP          4m6s

minikube service hello-minikube --url

http://192.168.49.2:30462
🏃  Starting tunnel for service hello-minikube.
❗  Because you are using a Docker driver on windows, the terminal needs to be open to run it.

I'm running in PowerShell 7.2.4 and this command failed since Get-Process doesn't have an -ef equivalent
ps -ef | grep docker@127.0.0.1

So I ran ps and here are the relevant results...

     16    21.37      22.77       0.80    2860   2 com.docker.backend
     28    30.02      37.50       4.80   11760   2 com.docker.backend
     13    17.41      13.59       0.08   18068   2 com.docker.dev-envs
     19    24.51      22.42       0.44    8336   2 com.docker.extensions
     17    24.84      24.11       2.47   20708   2 com.docker.proxy
     45    50.53      43.13       0.00    5820   0 com.docker.service
     15    18.23      17.15       0.22   21108   2 com.docker.wsl-distro-proxy
     11    22.96      22.62       0.25   17548   2 docker
     71    90.14      85.33       5.59    1800   2 Docker Desktop
     16    12.36      38.52       1.25   11628   2 Docker Desktop
     24    75.49      55.60       2.19   12240   2 Docker Desktop
     34    93.26      77.69       8.16   15920   2 Docker Desktop
     16    52.67      89.58       6.23   19552   2 Docker Desktop
     13    13.06      66.41       0.27   23064   2 Docker Desktop
      0 5,260.45   5,226.01       0.00   13196   0 vmmem
      7     1.35       6.32       0.02   11344   2 wsl
      7     1.35       6.33       0.03   19584   2 wsl
      7     1.35       6.44       0.03   19728   2 wsl
      7     1.43       6.42       1.59   20084   2 wsl
      7     1.34       6.45       0.02   20524   2 wsl
      7     1.17       6.03       0.03    3544   2 wslhost
      7     1.17       6.01       0.02   10412   2 wslhost
      7     1.25       6.38       0.03   19036   2 wslhost
      7     1.20       6.03       0.03   20072   2 wslhost
      7     1.20       6.03       0.02   20504   2 wslhost
      7     1.27       6.07       0.02   20840   2 wslhost

Leaving that terminal window open, navigating to http://192.168.49.2:30462 nor http://127.0.0.1:30462 nor http://localhost:30462 seems to work.

However, running kubectl port-forward service/hello-minikube 7080:8080 does work when I navigate to http://localhost:7080/

[deivydas321]

@BradKnowles thank you, it worked 🎉

[klaases]

Hi @anowak-ct, are you able to try the workaround that @BradKnowles suggested in his comment above?

[anowak-ct]

Hi @klaases, yes as I mentioned in my update comment, port forwarding works, however, using a tunnel or minikube service <svc> does not.

On an unrelated note, I am switching to the minikube hyper-v driver on windows. A colleague of mine face lots of issues with docker. hyper-v works better ...