Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Ingress on MacOS, driver docker #12089

Merged
merged 1 commit into from
Aug 20, 2021

Conversation

zhan9san
Copy link
Contributor

@zhan9san zhan9san commented Aug 1, 2021

Could you please kindly review this PR?

fixes #7332

How to use

  1. Enable nginx-ingress
    minikube addons enable ingress
    
  2. Create a service and ingress
    kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
    kubectl expose deployment web --port=8080
    kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
    
  3. Add the following line to the bottom of the /etc/hosts file.
    127.0.0.1 hello-world.info
    
  4. Create tunnel
    sudo permission will be asked for it. It needs password
    minikube tunnel
    
  5. Verify that the Ingress controller is directing traffic:
    curl hello-world.info
    

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 1, 2021
@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 1, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @zhan9san. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 1, 2021
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

return
}

resourcePorts := []int32{80}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

80, 443

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Thanks for your attention. I add port 443 as well.

But you may know the certificate problem below

$ curl https://hello-world.info
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
curl -k https://hello-world.info
Hello, world!
Version: 1.0.0
Hostname: web-79d88c97d6-xl5gv

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Thanks for taking my comment into consideration
The way I tested

# start new minikube with ingress addon
minikube start --nodes=2 --addons=ingress

# create workload
kubectl create deployment web --replicas=2 \
   --image=gcr.io/google-samples/hello-app:1.0

# create service
kubectl expose deployment web \
   --type=NodePort --port=8080

generate self-singed certificates for hello-world.info using this doc

# create tls secret
kubectl create secret tls hello-world-tls \
  --cert hello-world.info.pem \
  --key hello-world.info.key

# create ingress
kubectl create ingress hello-world-ingress \
  --rule=hello-world.info/=web:8080,tls=hello-world-tls

add 127.0.0.1 hello-world.info to /etc/hosts

start tunnel

minikube tunnel

check

curl https://hello-world.info/

and this worked

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm glad to know it worked for you.
Besides, it tried your openssl-script and it worked great as well.

@sharifelgamal
Copy link
Collaborator

Hi @zhan9san, thank you so much for this PR! This fixes a long standing issue in our codebase and it's greatly appreciated. Could you change the ingress test at https://github.com/kubernetes/minikube/blob/master/test/integration/addons_test.go#L146 so that it doesn't skip the test for darwin docker driver anymore?

@zhan9san
Copy link
Contributor Author

zhan9san commented Aug 13, 2021

so that it doesn't skip the test for darwin docker driver anymore

Currently, make integration will fail even when I use master branch.

I don't find integration test in Github workflow. Is it always running in local laptop?
After I enable ingress test on Darwin Docker driver, the integration tests for Ingress on MacOS pass as below.

$ make integration -e TEST_ARGS="-minikube-start-args=--driver=docker -test.run TestAddons/parallel/Ingress --profile=minikube"
go test -ldflags="-X k8s.io/minikube/pkg/version.version=v1.22.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.22.0-1627488369-11483 -X k8s.io/minikube/pkg/version.gitCommitID="6a21faddb60d2f950102b217d5385dcba82c2b79" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -v -test.timeout=90m ./test/integration --tags="integration " -minikube-start-args=--driver=docker -test.run TestAddons/parallel/Ingress --profile=minikube 2>&1 | tee "./out/testout_6a21faddb.txt"
Found 16 cores, limiting parallelism with --test.parallel=2
=== RUN   TestAddons
    addons_test.go:74: (dbg) Run:  out/minikube start -p minikube --wait=true --memory=4000 --alsologtostderr --addons=registry --addons=metrics-server --addons=olm --addons=volumesnapshots --addons=csi-hostpath-driver --driver=docker --addons=ingress --addons=helm-tiller --addons=gcp-auth
    addons_test.go:74: (dbg) Done: out/minikube start -p minikube --wait=true --memory=4000 --alsologtostderr --addons=registry --addons=metrics-server --addons=olm --addons=volumesnapshots --addons=csi-hostpath-driver --driver=docker --addons=ingress --addons=helm-tiller --addons=gcp-auth: (9m21.196964952s)
=== RUN   TestAddons/parallel
=== RUN   TestAddons/parallel/Ingress
=== PAUSE TestAddons/parallel/Ingress
=== CONT  TestAddons/parallel/Ingress
    addons_test.go:157: (dbg) TestAddons/parallel/Ingress: waiting 12m0s for pods matching "app.kubernetes.io/name=ingress-nginx" in namespace "ingress-nginx" ...
    helpers_test.go:340: "ingress-nginx-admission-create-6x2t8" [811ee9aa-e140-4eb0-8a7a-d74f7a37d58a] Succeeded: Initialized:PodCompleted / Ready:PodCompleted / ContainersReady:PodCompleted
    addons_test.go:157: (dbg) TestAddons/parallel/Ingress: app.kubernetes.io/name=ingress-nginx healthy within 8.279299ms
    addons_test.go:164: (dbg) Run:  kubectl --context minikube replace --force -f testdata/nginx-ingv1beta.yaml
    addons_test.go:164: (dbg) Done: kubectl --context minikube replace --force -f testdata/nginx-ingv1beta.yaml: (2.305915706s)
    addons_test.go:169: kubectl --context minikube replace --force -f testdata/nginx-ingv1beta.yaml: unexpected stderr: Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
         (may be temporary)
    addons_test.go:179: (dbg) Run:  kubectl --context minikube replace --force -f testdata/nginx-pod-svc.yaml
    addons_test.go:184: (dbg) TestAddons/parallel/Ingress: waiting 4m0s for pods matching "run=nginx" in namespace "default" ...
    helpers_test.go:340: "nginx" [a5c94789-bd8c-4339-b8dd-952f2be625ab] Pending
    helpers_test.go:340: "nginx" [a5c94789-bd8c-4339-b8dd-952f2be625ab] Pending / Ready:ContainersNotReady (containers with unready status: [nginx]) / ContainersReady:ContainersNotReady (containers with unready status: [nginx])
    helpers_test.go:340: "nginx" [a5c94789-bd8c-4339-b8dd-952f2be625ab] Running
    addons_test.go:184: (dbg) TestAddons/parallel/Ingress: run=nginx healthy within 40.02793691s
    addons_test.go:203: (dbg) Run:  out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'"
    addons_test.go:203: (dbg) Done: out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'": (1.083960193s)
    addons_test.go:229: (dbg) Run:  kubectl --context minikube replace --force -f testdata/nginx-ingv1.yaml
    addons_test.go:254: (dbg) Run:  out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'"
    addons_test.go:254: (dbg) Done: out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'": (1.02395979s)
    addons_test.go:254: (dbg) Run:  out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'"
    addons_test.go:254: (dbg) Done: out/minikube -p minikube ssh "curl -s http://127.0.0.1/ -H 'Host: nginx.example.com'": (1.817897408s)
    addons_test.go:277: (dbg) Run:  out/minikube -p minikube addons disable ingress --alsologtostderr -v=1
    addons_test.go:277: (dbg) Done: out/minikube -p minikube addons disable ingress --alsologtostderr -v=1: (29.171618017s)
=== CONT  TestAddons
    addons_test.go:128: (dbg) Run:  out/minikube stop -p minikube
    addons_test.go:128: (dbg) Done: out/minikube stop -p minikube: (7.457957415s)
    addons_test.go:132: (dbg) Run:  out/minikube addons enable dashboard -p minikube
    addons_test.go:136: (dbg) Run:  out/minikube addons disable dashboard -p minikube
    helpers_test.go:176: Cleaning up "minikube" profile ...
    helpers_test.go:179: (dbg) Run:  out/minikube delete -p minikube
    helpers_test.go:179: (dbg) Done: out/minikube delete -p minikube: (4.706535105s)
--- PASS: TestAddons (650.63s)
    --- PASS: TestAddons/parallel (0.00s)
        --- PASS: TestAddons/parallel/Ingress (76.73s)
PASS
Tests completed in 10m50.631593796s (result code 0)
ok  	k8s.io/minikube/test/integration	652.285s

As for the change in pkg/minikube/out/out.go, it fixes the unit test issue on MacOS which doesn't exist on Linux.

Error log is from this file, box-cli-maker, detect_unix.go and box-cli-maker, box.go

$ make test
...
open /proc/sys/kernel/osrelease: no such file or directory
...

The file, /proc/sys/kernel/osrelease doesn't exist on MacOS but Linux.

This issue results from the wrong value of box.Config.Color in

box.Config.Color = ""
and
isT := IsTerminal(w)

Here is an example to re-produce the issue, GoTerminal

The default value of box.Config.Color is RED, but it will be changed when we get a different value of IsTerminal.

That's why the Minikube binary file works well but integration/unit test fail.

@sharifelgamal
Copy link
Collaborator

so that it doesn't skip the test for darwin docker driver anymore

Currently, make integration will fail even when I use master branch.

I don't find integration test in Github workflow. Is it always running in local laptop?

yeah, not of all of our integration tests pass currently, it's something we're actively working on. the integration tests don't run in github actions, they run in jenkins after one of our maintainers mark the PR as ok to test.

@sharifelgamal
Copy link
Collaborator

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 13, 2021
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 47.8s    | 47.4s               |
| enable ingress | 31.5s    | 30.9s               |
+----------------+----------+---------------------+

Times for minikube (PR 12089) start: 47.9s 47.3s 47.5s 47.1s 47.1s
Times for minikube start: 48.5s 47.6s 47.6s 47.8s 47.5s

Times for minikube ingress: 32.4s 31.8s 31.8s 30.8s 30.8s
Times for minikube (PR 12089) ingress: 30.3s 30.3s 30.8s 32.3s 30.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 22.7s    | 22.1s               |
| enable ingress | 27.1s    | 28.4s               |
+----------------+----------+---------------------+

Times for minikube start: 23.4s 21.7s 22.4s 22.8s 23.1s
Times for minikube (PR 12089) start: 22.3s 22.4s 21.8s 22.3s 21.7s

Times for minikube ingress: 27.0s 27.5s 27.0s 27.0s 27.0s
Times for minikube (PR 12089) ingress: 27.1s 27.5s 34.5s 26.0s 27.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 40.2s    | 41.6s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 27.5s 43.3s 43.2s 43.4s 43.7s
Times for minikube (PR 12089) start: 44.3s 40.3s 42.5s 36.9s 44.0s

@sharifelgamal
Copy link
Collaborator

Hey @zhan9san, the tests look good! Once you rebase to fix the merge conflict, I'll go ahead and merge this PR. Thanks again for the contribution!

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 13, 2021
@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_crio_arm64 TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 2.00 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 2.45 (chart)
Docker_Linux_crio TestNetworkPlugins/group/bridge/DNS (gopogh) 4.35 (chart)
KVM_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 4.91 (chart)
KVM_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 4.91 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/embed-certs/serial/Pause (gopogh) 5.13 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/newest-cni/serial/Pause (gopogh) 5.13 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 5.16 (chart)
KVM_Linux_crio TestStartStop/group/newest-cni/serial/Pause (gopogh) 5.16 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/no-preload/serial/Pause (gopogh) 5.19 (chart)
KVM_Linux_crio TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 5.26 (chart)
KVM_Linux_containerd TestPause/serial/Pause (gopogh) 5.52 (chart)
KVM_Linux_containerd TestPause/serial/VerifyStatus (gopogh) 5.52 (chart)
KVM_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 5.52 (chart)
KVM_Linux_containerd TestStartStop/group/no-preload/serial/Pause (gopogh) 5.52 (chart)
KVM_Linux_crio TestStartStop/group/embed-certs/serial/Pause (gopogh) 5.81 (chart)
KVM_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 5.84 (chart)
Docker_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 5.88 (chart)
KVM_Linux_crio TestPause/serial/Pause (gopogh) 5.88 (chart)
KVM_Linux_crio TestPause/serial/VerifyStatus (gopogh) 5.88 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 6.17 (chart)
KVM_Linux_crio TestStartStop/group/no-preload/serial/Pause (gopogh) 7.14 (chart)
Docker_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 7.41 (chart)
Docker_Linux_crio TestStartStop/group/newest-cni/serial/Pause (gopogh) 8.00 (chart)
Docker_Linux TestStartStop/group/embed-certs/serial/Pause (gopogh) 8.64 (chart)
Docker_Linux_crio TestStartStop/group/no-preload/serial/Pause (gopogh) 8.87 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/AddonExistsAfterStop (gopogh) 8.87 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 8.87 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 8.87 (chart)
Docker_Windows TestStartStop/group/default-k8s-different-port/serial/VerifyKubernetesImages (gopogh) 8.87 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 14, 2021
@zhan9san
Copy link
Contributor Author

Hi @shahiddev
The conflict has been fixed.
Could you please review it again?

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 49.0s    | 48.6s               |
| enable ingress | 30.8s    | 31.4s               |
+----------------+----------+---------------------+

Times for minikube start: 49.5s 48.8s 46.5s 48.2s 51.8s
Times for minikube (PR 12089) start: 50.5s 51.2s 47.2s 47.0s 47.3s

Times for minikube ingress: 31.3s 30.3s 30.8s 30.8s 30.8s
Times for minikube (PR 12089) ingress: 30.4s 30.8s 32.3s 32.2s 31.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 22.4s    | 22.2s               |
| enable ingress | 26.9s    | 28.3s               |
+----------------+----------+---------------------+

Times for minikube (PR 12089) ingress: 27.0s 28.0s 25.5s 26.5s 34.6s
Times for minikube ingress: 27.5s 27.0s 27.0s 27.0s 26.0s

Times for minikube start: 23.0s 21.2s 21.3s 23.4s 23.0s
Times for minikube (PR 12089) start: 21.9s 21.7s 23.0s 22.4s 22.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 41.0s    | 43.4s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 28.1s 44.8s 43.5s 44.4s 44.0s
Times for minikube (PR 12089) start: 42.9s 42.8s 43.6s 43.5s 44.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
none_Linux TestFunctional/serial/ComponentHealth (gopogh) 3.16 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 4.49 (chart)
Docker_Linux_crio TestNetworkPlugins/group/bridge/DNS (gopogh) 4.52 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/AddonExistsAfterStop (gopogh) 6.96 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 6.96 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/VerifyKubernetesImages (gopogh) 6.96 (chart)
KVM_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 7.69 (chart)
KVM_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 7.69 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 7.76 (chart)
KVM_Linux_crio TestStartStop/group/newest-cni/serial/Pause (gopogh) 8.05 (chart)
KVM_Linux_crio TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 8.22 (chart)
KVM_Linux_containerd TestPause/serial/Pause (gopogh) 8.33 (chart)
KVM_Linux_containerd TestPause/serial/VerifyStatus (gopogh) 8.33 (chart)
KVM_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 8.33 (chart)
KVM_Linux_containerd TestStartStop/group/no-preload/serial/Pause (gopogh) 8.33 (chart)
Docker_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 8.70 (chart)
KVM_Linux_crio TestStartStop/group/embed-certs/serial/Pause (gopogh) 8.72 (chart)
KVM_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 8.78 (chart)
KVM_Linux_crio TestPause/serial/Pause (gopogh) 8.84 (chart)
KVM_Linux_crio TestPause/serial/VerifyStatus (gopogh) 8.84 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 9.03 (chart)
Docker_Linux TestStartStop/group/embed-certs/serial/Pause (gopogh) 9.03 (chart)
KVM_Linux_crio TestStartStop/group/no-preload/serial/Pause (gopogh) 9.46 (chart)
Docker_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 10.32 (chart)
Docker_Linux_crio TestStartStop/group/newest-cni/serial/Pause (gopogh) 11.03 (chart)
Docker_Linux_crio TestStartStop/group/no-preload/serial/Pause (gopogh) 11.67 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/Pause (gopogh) 12.26 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 12.90 (chart)
Docker_Windows TestStartStop/group/embed-certs/serial/Pause (gopogh) 13.91 (chart)
Docker_Linux_crio TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 14.10 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@zhan9san
Copy link
Contributor Author

Hi @shahiddev
The conflict has been fixed.
Could you please review it again?

Sorry, it seems the conflicts are not fixed completely. I'll fix it these days

@zhan9san zhan9san force-pushed the feature/ingress-mac branch from da5c26e to 22bf286 Compare August 15, 2021 05:35
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 15, 2021
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 48.2s    | 47.7s               |
| enable ingress | 33.1s    | 32.3s               |
+----------------+----------+---------------------+

Times for minikube start: 48.8s 46.5s 48.0s 46.3s 51.5s
Times for minikube (PR 12089) start: 48.1s 46.8s 48.4s 49.3s 46.1s

Times for minikube ingress: 33.3s 33.7s 32.7s 33.8s 32.2s
Times for minikube (PR 12089) ingress: 31.2s 31.8s 33.3s 32.8s 32.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 21.9s    | 21.4s               |
| enable ingress | 35.0s    | 35.8s               |
+----------------+----------+---------------------+

Times for minikube ingress: 38.0s 32.5s 32.0s 35.5s 37.0s
Times for minikube (PR 12089) ingress: 35.5s 31.5s 35.5s 39.0s 37.5s

Times for minikube start: 23.5s 21.2s 22.0s 21.8s 21.1s
Times for minikube (PR 12089) start: 20.7s 22.2s 21.7s 21.3s 21.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 41.5s    | 43.1s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube (PR 12089) start: 43.3s 43.0s 43.3s 44.2s 41.6s
Times for minikube start: 31.9s 43.8s 44.4s 43.7s 43.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Windows TestKicCustomNetwork/use_default_bridge_network (gopogh) 0.00 (chart)
Docker_Windows TestKicExistingNetwork (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/AddNode (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/DeleteNode (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/DeployApp2Nodes (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/FreshStart2Nodes (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/ProfileList (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/StartAfterStop (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/StopNode (gopogh) 0.00 (chart)
KVM_Linux_containerd TestFunctional/parallel/MountCmd/specific-port (gopogh) 0.00 (chart)
Docker_Windows TestMultiNode/serial/StopMultiNode (gopogh) 0.72 (chart)
Docker_Windows TestForceSystemdEnv (gopogh) 0.74 (chart)
Docker_Windows TestPreload (gopogh) 1.44 (chart)
Docker_Windows TestForceSystemdFlag (gopogh) 1.50 (chart)
Docker_Windows TestDockerFlags (gopogh) 2.96 (chart)
Docker_Windows TestPause/serial/SecondStartNoReconfiguration (gopogh) 3.10 (chart)
KVM_Linux_crio TestAddons/parallel/CSI (gopogh) 3.38 (chart)
Docker_Linux_crio TestKubernetesUpgrade (gopogh) 4.46 (chart)
Docker_Linux_crio_arm64 TestKubernetesUpgrade (gopogh) 5.76 (chart)
Docker_Windows TestOffline (gopogh) 5.76 (chart)
Docker_Linux_crio TestNetworkPlugins/group/auto/DNS (gopogh) 7.01 (chart)
Docker_Windows TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 7.91 (chart)
Docker_Windows TestMultiNode/serial/RestartMultiNode (gopogh) 7.97 (chart)
Docker_Windows TestFunctional/parallel/InternationalLanguage (gopogh) 9.35 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 10.19 (chart)
Docker_Windows TestKubernetesUpgrade (gopogh) 10.53 (chart)
Docker_Linux_crio_arm64 TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 10.99 (chart)
Docker_Linux_crio_arm64 TestStartStop/group/no-preload/serial/EnableAddonWhileActive (gopogh) 10.99 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux_crio TestStartStop/group/newest-cni/serial/SecondStart (gopogh) 0.00 (chart)
KVM_Linux_containerd TestAddons/parallel/GCPAuth (gopogh) 0.62 (chart)
Docker_Linux TestAddons/parallel/Olm (gopogh) 1.25 (chart)
Docker_Linux_docker_arm64 TestMultiNode/serial/DeleteNode (gopogh) 7.06 (chart)
Docker_Linux_docker_arm64 TestMultiNode/serial/RestartMultiNode (gopogh) 7.06 (chart)
Docker_Linux_docker_arm64 TestMultiNode/serial/RestartKeepsNodes (gopogh) 10.47 (chart)
Hyperkit_macOS TestFunctional/parallel/DashboardCmd (gopogh) 14.17 (chart)
Docker_Linux_docker_arm64 TestAddons/parallel/CSI (gopogh) 15.05 (chart)
Docker_Linux_docker_arm64 TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 16.13 (chart)
Docker_Linux_docker_arm64 TestMultiNode/serial/FreshStart2Nodes (gopogh) 16.13 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 19.25 (chart)
Docker_Linux_crio TestStartStop/group/no-preload/serial/Stop (gopogh) 20.62 (chart)
Docker_Linux_docker_arm64 TestFunctional/parallel/PersistentVolumeClaim (gopogh) 22.58 (chart)
Docker_Linux_docker_arm64 TestFunctional/parallel/MountCmd/any-port (gopogh) 23.66 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/embed-certs/serial/Pause (gopogh) 23.84 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/newest-cni/serial/Pause (gopogh) 23.84 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 24.00 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/no-preload/serial/Pause (gopogh) 24.16 (chart)
KVM_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 24.22 (chart)
KVM_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 24.22 (chart)
KVM_Linux_containerd TestPause/serial/Pause (gopogh) 24.84 (chart)
KVM_Linux_containerd TestPause/serial/VerifyStatus (gopogh) 24.84 (chart)
KVM_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 24.84 (chart)
KVM_Linux_containerd TestStartStop/group/no-preload/serial/Pause (gopogh) 24.84 (chart)
KVM_Linux_crio TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 25.17 (chart)
KVM_Linux_crio TestStartStop/group/newest-cni/serial/Pause (gopogh) 25.32 (chart)
KVM_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 25.49 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 25.62 (chart)
KVM_Linux_crio TestStartStop/group/embed-certs/serial/Pause (gopogh) 25.97 (chart)
Docker_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 26.25 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 16, 2021
@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd_arm64 TestNetworkPlugins/group/calico/NetCatPod (gopogh) 0.00 (chart)
Docker_Linux TestAddons/parallel/Olm (gopogh) 1.32 (chart)
Docker_macOS TestNetworkPlugins/group/cilium/Start (gopogh) 1.96 (chart)
Docker_Linux_crio TestKubernetesUpgrade (gopogh) 5.30 (chart)
Docker_Linux_crio TestNetworkPlugins/group/kindnet/DNS (gopogh) 8.61 (chart)
Docker_Linux TestFunctional/serial/ComponentHealth (gopogh) 11.26 (chart)
Docker_macOS TestStartStop/group/embed-certs/serial/Pause (gopogh) 14.00 (chart)
Docker_macOS TestDebPackageInstall/install_amd64_debian (gopogh) 10/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_debian (gopogh) 9/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_debian (gopogh) latest/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_debian (gopogh) sid/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_ubuntu (gopogh) 18.04/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_ubuntu (gopogh) 20.04/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_ubuntu (gopogh) 20.10/kvm2-driver (chart)
Docker_macOS TestDebPackageInstall/install_amd64_ubuntu (gopogh) latest/kvm2-driver (chart)
Docker_Linux_containerd_arm64 TestAddons/parallel/Ingress (gopogh) 21.38 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 21.71 (chart)
Docker_Linux_containerd_arm64 TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 22.76 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/embed-certs/serial/Pause (gopogh) 26.57 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/newest-cni/serial/Pause (gopogh) 26.57 (chart)
Docker_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 26.61 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 26.76 (chart)
Docker_Linux_containerd_arm64 TestStartStop/group/no-preload/serial/Pause (gopogh) 26.95 (chart)
KVM_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 26.97 (chart)
KVM_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 26.97 (chart)
Docker_macOS TestNetworkPlugins/group/bridge/Start (gopogh) 27.45 (chart)
Docker_Linux_containerd_arm64 TestAddons/parallel/CSI (gopogh) 27.59 (chart)
KVM_Linux_containerd TestPause/serial/Pause (gopogh) 27.63 (chart)
KVM_Linux_containerd TestPause/serial/VerifyStatus (gopogh) 27.63 (chart)
KVM_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 27.63 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@zhan9san zhan9san force-pushed the feature/ingress-mac branch from 22bf286 to def3f06 Compare August 17, 2021 03:13
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2021
@zhan9san
Copy link
Contributor Author

I performed another rebase operation. After that,

  1. make test failed but all checks passed before rebase

    $ make test
    coverage: 55.4% of statements
    FAIL	k8s.io/minikube/pkg/minikube/out/register	11.381s
    ok  	k8s.io/minikube/pkg/minikube/perf	15.387s	coverage: 21.2% of statements
    ok  	k8s.io/minikube/pkg/minikube/proxy	11.121s	coverage: 68.7% of statements
    ok  	k8s.io/minikube/pkg/minikube/reason	11.496s	coverage: 70.0% of statements
    ok  	k8s.io/minikube/pkg/minikube/registry	10.973s	coverage: 77.0% of statements
    ok  	k8s.io/minikube/pkg/minikube/registry/drvs/docker	10.204s	coverage: 19.8% of statements
    ok  	k8s.io/minikube/pkg/minikube/registry/drvs/hyperkit	9.301s	coverage: 41.5% of statements
    ok  	k8s.io/minikube/pkg/minikube/service	10.356s	coverage: 84.2% of statements
    ok  	k8s.io/minikube/pkg/minikube/shell	9.425s	coverage: 94.3% of statements
    ok  	k8s.io/minikube/pkg/minikube/storageclass	9.598s	coverage: 100.0% of statements
    ok  	k8s.io/minikube/pkg/minikube/style	9.577s	coverage: 100.0% of statements
    ok  	k8s.io/minikube/pkg/minikube/sysinit	9.723s	coverage: 4.5% of statements
    ok  	k8s.io/minikube/pkg/minikube/translate	10.664s	coverage: 45.5% of statements
    ok  	k8s.io/minikube/pkg/minikube/tunnel	11.940s	coverage: 57.4% of statements
    ok  	k8s.io/minikube/pkg/util	9.776s	coverage: 75.7% of statements
    ok  	k8s.io/minikube/pkg/util/lock	9.638s	coverage: 22.2% of statements
    ok  	k8s.io/minikube/pkg/util/retry	9.519s	coverage: 0.0% of statements
    FAIL
    make: *** [test] Error 32
    
  2. make integration -e TEST_ARGS="-minikube-start-args=--driver=docker -test.run TestAddons/parallel/Ingress --profile=minikube" failed but it worked well before rebase.

    Fortunately, make integration -e TEST_ARGS="-minikube-start-args=--driver=docker -test.run TestAddons" works

Concerns:

  1. I run make test on master branch, and it failed as well. It seems the CI gateway doesn't block PR in which unit test failed.
  2. It seems the incoming code does not comply with quickly-iterating-on-a-single-test rule. Should someone update the document ?

Because I don't have much experience in Golang, please correct me if I say something wrong.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 51.8s    | 50.2s               |
| enable ingress | 34.7s    | 39.5s               |
+----------------+----------+---------------------+

Times for minikube start: 53.6s 52.6s 49.7s 51.4s 51.7s
Times for minikube (PR 12089) start: 49.2s 48.0s 50.8s 49.1s 53.8s

Times for minikube (PR 12089) ingress: 40.5s 41.9s 33.4s 40.4s 41.5s
Times for minikube ingress: 34.9s 31.9s 41.4s 32.4s 32.9s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 23.5s    | 23.3s               |
| enable ingress | 33.1s    | 35.8s               |
+----------------+----------+---------------------+

Times for minikube start: 24.5s 23.1s 23.8s 23.0s 22.9s
Times for minikube (PR 12089) start: 23.6s 23.1s 23.7s 22.6s 23.5s

Times for minikube (PR 12089) ingress: 35.6s 36.1s 36.1s 35.6s 35.6s
Times for minikube ingress: 32.1s 32.6s 31.6s 31.6s 37.6s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12089) |
+----------------+----------+---------------------+
| minikube start | 38.3s    | 44.3s               |
| enable ingress |          |                     |
+----------------+----------+---------------------+

Times for minikube start: 31.5s 27.1s 44.6s 44.6s 43.6s
Times for minikube (PR 12089) start: 45.1s 41.2s 47.3s 44.2s 44.0s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_macOS TestAddons/parallel/CSI (gopogh) n/a
Docker_macOS TestFunctional/parallel/MySQL (gopogh) 0.00 (chart)
Docker_macOS TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 0.00 (chart)
Docker_macOS TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh) 0.00 (chart)
KVM_Linux_crio TestScheduledStopUnix (gopogh) 0.67 (chart)
Docker_Windows TestMultiNode/serial/StopMultiNode (gopogh) 0.74 (chart)
KVM_Linux_containerd TestAddons/parallel/CSI (gopogh) 3.21 (chart)
Docker_Linux_containerd TestKubernetesUpgrade (gopogh) 4.52 (chart)
Docker_macOS TestFunctional/parallel/PersistentVolumeClaim (gopogh) 5.56 (chart)
none_Linux TestFunctional/serial/LogsCmd (gopogh) 6.25 (chart)
Docker_macOS TestFunctional/parallel/MountCmd/any-port (gopogh) 7.41 (chart)
Docker_Linux_crio_arm64 TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 7.87 (chart)
Docker_Linux_crio TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 8.39 (chart)
Docker_macOS TestMultiNode/serial/AddNode (gopogh) 9.26 (chart)
Docker_Windows TestKubernetesUpgrade (gopogh) 10.16 (chart)
Docker_Windows TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh) 10.37 (chart)
Docker_Windows TestFunctional/parallel/InternationalLanguage (gopogh) 11.11 (chart)
Docker_Linux_crio_arm64 TestFunctional/parallel/BuildImage (gopogh) 11.43 (chart)
Docker_Windows TestAddons/parallel/HelmTiller (gopogh) 20.74 (chart)
KVM_Linux_containerd TestStartStop/group/newest-cni/serial/Pause (gopogh) 23.08 (chart)
Docker_Windows TestInsufficientStorage (gopogh) 26.67 (chart)
Docker_Linux_crio TestStartStop/group/default-k8s-different-port/serial/Stop (gopogh) 27.74 (chart)
KVM_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/Pause (gopogh) 28.85 (chart)
KVM_Linux_containerd TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 28.85 (chart)
KVM_Linux_containerd TestPause/serial/Pause (gopogh) 29.49 (chart)
KVM_Linux_containerd TestPause/serial/VerifyStatus (gopogh) 29.49 (chart)
KVM_Linux_containerd TestStartStop/group/embed-certs/serial/Pause (gopogh) 29.49 (chart)
KVM_Linux_containerd TestStartStop/group/no-preload/serial/Pause (gopogh) 29.49 (chart)
KVM_Linux_crio TestStartStop/group/old-k8s-version/serial/Pause (gopogh) 29.93 (chart)
Docker_Linux_crio_arm64 TestPause/serial/Pause (gopogh) 30.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@zhan9san
Copy link
Contributor Author

Hi @sharifelgamal
Sorry to bother you.
Could you kindly review this PR?

@sharifelgamal
Copy link
Collaborator

It seems ok to me, and the tests pass, I just want another pair of eyes before we merge it.

cc @medyagh

@medyagh
Copy link
Member

medyagh commented Aug 20, 2021

thank you @zhan9san looks good to me

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, zhan9san

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2021
@sharifelgamal sharifelgamal merged commit 69d5d34 into kubernetes:master Aug 20, 2021
bradbeck added a commit to buildsec/frsca that referenced this pull request Feb 17, 2022
The referenced issue (kubernetes/minikube#7332)
appears to have been resolved by kubernetes/minikube#12089
kodiakhq bot added a commit to buildsec/frsca that referenced this pull request Feb 28, 2022
The referenced issue (kubernetes/minikube#7332)
appears to have been resolved by kubernetes/minikube#12089

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
@Ruitjes
Copy link

Ruitjes commented May 9, 2022

@zhan9san Are you required to add the IP at the bottom of the /etc/hosts file? or can you also skip that part and just curl 127.0.0.1?

@zhan9san
Copy link
Contributor Author

zhan9san commented May 9, 2022

@Ruitjes

I am sorry, and I can't follow you.

Does #13806 clear your concern?

If not, could you provide more detailed information?

@Findaa
Copy link

Findaa commented May 25, 2022

Hello. I think issue is still present. I followed 13788 discussion.
Monterey 12.3.1 Intel i7
This was reproduced on docker driver too.

minikube start --driver=hyperkit
😄  minikube v1.25.2 on Darwin 12.3.1
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
deployment.apps/web created
kubectl expose deployment web --port=8080
service/web exposed
kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
ingress.networking.k8s.io/example-ingress created`
cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
127.0.0.1 hello-world.info
# End of section
minikube tunnel --cleanup
Status:
	machine: minikube
	pid: 65900
	route: 10.96.0.0/12 -> 192.168.64.4
	minikube: Running
	services: []
    errors:
		minikube: no errors
		router: no errors
		loadbalancer emulator: no errors
curl hello-world.info

is not anyhow responding and is stuck until quit.

I was looking into 13806 and as that is a fresh PR wanted to confirm if that is patchable at that moment.

Also, I noticed this comment about alternate addresses.

kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE    SELECTOR
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    7m9s   <none>
web          ClusterIP   10.105.66.48   <none>        8080/TCP   6m9s   app=web
ps -ef | grep ssh
  501  1472     1   0 Thu03AM ??         0:00.07 /usr/bin/ssh-agent -l
    0 34334     1   0 Mon08PM ??         0:00.03 sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxx/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 34653 34334   0 Mon08PM ??         0:00.01 ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxx/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 60910     1   0 Tue02AM ??         0:00.03 sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxx/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
  501 71488 69063   0  2:58AM ttys005    0:00.00 grep ssh

I tried grepping cluster ip:

ps -ef | grep 10.105.66.48
  501 73767 69063   0  3:00AM ttys005    0:00.00 grep 10.105.66.48

Looks like ps is missing here for cluster ip, grep of cluster ip returns a record but without port. Is that a possible reason and if so what would be the fix?

I was able to access bitnami postgres by both other service and by dbeaver using minikube tunneling so I think that is an ingress issue.

Also addressing already mentioned comment, this neither does help:

kubectl -n kube-system rollout restart deployment coredns
deployment.apps/coredns restarted
curl hello-world.info


@zhan9san
Copy link
Contributor Author

@Findaa

minikube start --driver=hyperkit

It seems your are not using Docker driver.

@Findaa
Copy link

Findaa commented May 25, 2022

@zhan9san
I was using it too. I noticed about it in prev comment, but here you can see the steps if that helps debugging:

xxx:~ xxxoo$ minikube delete
🔥  Deleting "minikube" in hyperkit ...
💀  Removed all traces of the "minikube" cluster.
xxx:~ xxxoo$ minikube start --driver=docker
😄  minikube v1.25.2 on Darwin 12.3.1
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
xxx:~ xxxoo$ minikube addons enable ingress
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled
xxx:~ xxxoo$ kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
deployment.apps/web created
xxx:~ xxxoo$ kubectl expose deployment web --port=8080
service/web exposed
xxx:~ xxxoo$ kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
ingress.networking.k8s.io/example-ingress created
xxx:~ xxxoo$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
127.0.0.1 hello-world.info
# End of section

Here i run minikube tunnel on other tab, provide pass and confirm with enter.

xxx:~ xxxoo$ minikube tunnel --cleanup
✅  Tunnel successfully started

📌  NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...

❗  The service/ingress example-ingress requires privileged ports to be exposed: [80 443]
🔑  sudo permission will be asked for it.
🏃  Starting tunnel for service example-ingress.
Password:
xxx:~ xxxoo$ curl hello-world.info

xxx:~ xxxoo$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    5m47s
web          ClusterIP   10.109.64.59   <none>        8080/TCP   3m6s
xxx:~ xxxoo$ ps -ef | grep ssh
  501  1472     1   0 Thu03AM ??         0:00.08 /usr/bin/ssh-agent -l
    0 34334     1   0 Mon08PM ??         0:00.03 sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxxoo/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 34653 34334   0 Mon08PM ??         0:00.01 ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxxoo/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 60910     1   0 Tue02AM ??         0:00.03 sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 65017 -i /Users/xxxoo/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 42128 42110   0 10:33AM ttys003    0:00.03 sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 64329 -i /Users/xxxoo/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
    0 42129 42128   0 10:33AM ttys003    0:00.01 ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N docker@127.0.0.1 -p 64329 -i /Users/xxxoo/.minikube/machines/minikube/id_rsa -L 80:127.0.0.1:80 -L 443:127.0.0.1:443
  501 44720 69063   0 10:36AM ttys005    0:00.00 grep ssh

xxx:~ xxxoo$ kubectl -n kube-system rollout restart deployment coredns
deployment.apps/coredns restarted
xxx:~ xxxoo$ curl hello-world.info

@zhan9san
Copy link
Contributor Author

Hi @Findaa

Thanks for your detailed explanation.

Could you try to kill all other obsolete ssh process manually? and try again?

No need to recreate the service and ingress.

Based on the log above,

  1. Kill obsolete ssh process
sudo kill -9 34334 34653 60910 42128 42129
  1. Restart tunnel

Press Ctrl + c to stop the existing tunnel.

minikube tunnel --cleanup
  1. Verify service
$ curl hello-world.info
Hello, world!
Version: 1.0.0
Hostname: web-746c8679d4-9rt96

@Findaa
Copy link

Findaa commented May 25, 2022

Amazing, that helped. Thank you a lot @zhan9san that is working solution.
For people following thread and having same issue, pid's are rather rapidly changing so verify with ps.
Would it be possible to know what caused the obselote ssh services? I suppose with each minikube restart this is going to happen over again, but at least solution is not complicated.

@zhan9san
Copy link
Contributor Author

The root cause is that the 80 port is already bound which will result in unexpected behavior of minikube tunnel.

For your case, it's obsolete ssh process, maybe it's Nginx, Apache or other process for others.

Would it be possible to know what caused the obselote ssh services? I suppose with each minikube restart this is going to happen over again, but at least solution is not complicated.

I can't give you an exact answer. But I believe you know how to debug and fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

docker: Ingress not exposed on MacOS
10 participants