hyperkit: VM is unable to access k8s.gcr.io (when VPN is in use) #6296
Comments
|
@massenz sorry that you face this issue, I am curious does and also do you happen to use VPN or corp proxy? |
medyagh
added
triage/needs-information
|
Yes, Docker works just fine (with/without login) Yep, the BTW - this seems to be a good place to remark how awesome |
|
At last it seems this issue is with the hyperkitVM, because It works fine with the virtual box.
|
|
With minikube on virtualbox, all good till the deployment. My Problem is, pods running NodeJS service can't connect to the external services (e.g. MongoDB Atlas). Note: I have deployed on DigitalOcen, the same docker build work perfectly fine. 2020-02-24 17:05:02 [ info ] : server started on port 3333 (local) |
|
For hyperkit, Do you mind sharing the output of the following two commands for me?
Thank you! |
tstromberg
changed the title
tstromberg
changed the title
|
Minikube then runs without issues. If you want to keep |
|
Same problem here, no dnsmasq installed, when using HyperKit. The question is, which application does interfere on my machine? MacOs Mojave: 10.14.6 (18G2022) minikube version kubectl version |
|
I'm also running into this issue. Note: I also have AnyConnect running -- seems like this could be a common denominator. minikube version: v1.9.2 I've tried re-installing with, I'm running Docker Desktop 2.2.0.5 Following DNS Debugging, I see the following. |
|
Hi team I am using Ciso VPN, no matter i set the proxy or not, the result is the same. Can get the dns log I can get the some docker images inside the VM But when i try to login the dockerhub, failed Some article said it has relationship with the DNS After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this? By the way, |
|
I'm also seeing this, trying to run on a Mac that has a "Cisco Anconnect" VPN software and hyperkit .. the easiest workaround is to use the --vm-driver=virtualbox option. I'm happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not "turned on" to ensure I'm not accessing "inappropriate" websites like say .. urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it). |
I am using win10 pro and I have default hyper-v and cisco anyconnect how can we turn off hyper-v on windows and start using --vm-driver=virtualbox |
I have the probelm, can you please tell me how to get inside minikube vm? |
|
You can use minikube ssh
…On Thu, Apr 30, 2020, 7:36 PM mjm19091979 ***@***.***> wrote:
Hi team
I met the same error.
Mac Mojave , version 10.14.6. run the command minikube start
--vm-driver=hyperkit to start the minikube, below is the logs
😄 minikube v1.9.2 on Darwin 10.14.6
✨ Using the hyperkit driver based on existing profile
👍 Starting control plane node m01 in cluster minikube
🔄 Restarting existing hyperkit VM for "minikube" ...
🐳 Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗ This VM is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟 Enabling addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"
I am using Ciso VPN, no matter i set the proxy or not, the result is the
same.
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-fnxht 1/1 Running 5 56m
coredns-66bff467f8-vxr5s 1/1 Running 5 56m
etcd-minikube 1/1 Running 5 56m
kube-apiserver-minikube 1/1 Running 5 56m
kube-controller-manager-minikube 1/1 Running 5 56m
kube-proxy-pt5js 1/1 Running 5 56m
kube-scheduler-minikube 1/1 Running 5 56m
storage-provisioner 1/1 Running 8 56m
Can get the dns log
kubectl -n kube-system logs coredns-66bff467f8-fnxht
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" ***@***.***/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463 1 reflector.go:153] ***@***.***/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" ***@***.***/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753 1 reflector.go:153] ***@***.***/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236 1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" ***@***.***/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287 1 reflector.go:153] ***@***.***/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I can get the some docker images inside the VM
minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.18.0 43940c34f24f 4 weeks ago 117MB
k8s.gcr.io/kube-apiserver v1.18.0 74060cea7f70 4 weeks ago 173MB
k8s.gcr.io/kube-controller-manager v1.18.0 d3e55153f52f 4 weeks ago 162MB
k8s.gcr.io/kube-scheduler v1.18.0 a31f78c7c8ce 4 weeks ago 95.3MB
kubernetesui/dashboard v2.0.0-rc6 cdc71b5a8a0e 5 weeks ago 221MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 2 months ago 683kB
k8s.gcr.io/coredns 1.6.7 67da37a9a360 2 months ago 43.8MB
kindest/kindnetd 0.5.3 aa67fec7d7ef 5 months ago 78.5MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 6 months ago 288MB
kubernetesui/metrics-scraper v1.0.2 3b08661dc379 6 months ago 40.1MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb10 2 years ago 80.8MB
But when i try to login the dockerhub, failed
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Some article said it has relationship with the DNS
sudo lsof -i4UDP:53 -P -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder 33u IPv4 0x1154dc77691251b5 0t0 UDP *:53
After kill the process, error still occurs. It seems a bug for minikube
with hyperkit driver. How to fix this?
By the way,
minikube version
minikube version: v1.9.2
commit: 93af9c1
I have the probelm, can you please tell me how to get inside minikube vm?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#6296 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABC2KAZUETHTFA3DC4KGUPLRPIYRZANCNFSM4KGIXIMQ>
.
|
tstromberg
changed the title
tstromberg
added
the
needs-solution-message
|
We should update the error string to mention trying |
|
we need a solution message, if can't pull images due to network issues, only on windows and macos, if they are not already using docker driver they should use docker driver. |
medyagh
added
good first issue
sharifelgamal
added
priority/backlog
|
Running minikube alongside vpnkit on Mac seems to work, with a couple of gotchas. Building vpnkit from source fails on original's repo (moby/vpnkit) and latest binaries are not available anywhere. In short, the Makefile for Mac build needs a bit of tweaking for opam dependencies. So I forked the original repo to build from source: # install build dependencies
brew install opam gpatch pkg-config dune dylibbundler libtool automake
# build vpnkit
git clone git@github.com:ar2pi/vpnkit.git
cd vpnkit
make -f Makefile.darwin ocaml
make -f Makefile.darwin depends
make -f Makefile.darwin build
cp ~/.opam/4.12.0/bin/vpnkit /usr/local/bin/vpnkitThen hyperkit Homebrew's install also has a known issue, so we need to build that from source as well. # build hyperkit
brew uninstall hyperkit
git clone git@github.com:moby/hyperkit.git
cd hyperkit
make
cp build/hyperkit /usr/local/bin/hyperkitOnce you have vpnkit and hyperkit, you can run: # terminal 1
vpnkit --ethernet=/tmp/vpnkit.eth.sock
# terminal 2
minikube start --driver hyperkit --hyperkit-vpnkit-sock=/tmp/vpnkit.eth.sock --memory 8192 --cpus 4
eval $(minikube -p minikube docker-env)
# [...] your docker commandsAnd voilà! Docker will run within minikube's hyperkit VM, through vpnkit. But there's still a couple of connection error messages that have been bugging me for a few days when starting / restarting a new VM: And on vpnkit's output we can see: Once VM is started everything appears to be fine though, could pull a few images and run basic docker commands. Haven't yet tested container to container networking, nor file mounts. |
|
I just workaround by installing proxyman in my local laptop |
What are you setting your HTTP_PROXY to? |
After turning on VPN, I can't pull image because can't connect to the internet export HTTPS_PROXY=192.168.64.1:9090 and the postman will start a proxy and listen on port 9090 Reference to https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ |
|
Thanks! |
sharifelgamal
added
priority/important-soon
|
I tried long and hard to get minikube with hyperkit to work on macos I couldn't get past ... and related, in a persistent way. My corporate VPN means I have a bunch of cisco processes - even when the VPN is not running I tried unsuccessfully using File Sync to copy an I found that /etc/resolv.conf on the minikube VM got regularly overridden. I tried unsuccessfully to use a systemd dropin. I gave up and installed virtualbox instead. It worked out of the box, with and without VPN connected. |
|
I want to use minikube without docker desktop on mac by hyperkit vm. Also, I'm using VPN. I have tried setting proxy and VPN, refer to https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/, but not working. Any update on this issue? The minikube version: v1.25.2. Thanks |
|
I dont have a VPN (no vpn installed ever). I get This VM is having trouble accessing https://k8s.gcr.io using hyper kit driver, any hope this can be resolved or is hyperkit driver no longer supported? |
|
Could https://github.com/containers/gvisor-tap-vsock be of help here? edit:
|
|
BTW k8s.gcr.io is deprecated as a source of container images |
|
On my machine is some corporate software (cisco security, vpn) running and I got it working by starting |
|
This is still an issue in April 2023. I do not have a VPN running (direct internet connection) and I see: 😄 minikube v1.30.1 on Darwin 12.6.5 |
|
k8s.gcr.io is deprecated - see https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/ and other announcements. /remove-help |
k8s-ci-robot
removed
the
help wanted
|
@massenz would you be willing to revise this to provide an updated steps-to-reproduce, covering the new registry etc? |
and others
Starting
minikubewith the default VM driver (Hyperkit) makes the external network unreachable:stderr:
W0113 20:47:22.189603 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0113 20:47:22.190387 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0113 20:47:22.192219 2751 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0113 20:47:22.192254 2751 validation.go:28] Cannot validate kubelet config - no validator is available
failed to pull image "k8s.gcr.io/kube-apiserver:v1.17.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher
And then trying to deploy any pod fails.
Starting with
--vmdriver virtualboxworks just fine.The exact command to reproduce the issue:
utils.yaml pulls in
massenz/dnsutils:1.1image (but this is reproducible with any Docker image).The full output of the command that failed:
Normal BackOff 30s kubelet, minikube Back-off pulling image "massenz/dnsutils:1.1"
Warning Failed 30s kubelet, minikube Error: ImagePullBackOff
Normal Pulling 16s (x2 over 46s) kubelet, minikube Pulling image "massenz/dnsutils:1.1"
Warning Failed 1s (x2 over 31s) kubelet, minikube Failed to pull image "massenz/dnsutils:1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 1s (x2 over 31s) kubelet, minikube Error: ErrImagePull
The output of the
minikube logscommand:Jan 11 14:19:23 minikube kubelet[4757]: W0111 14:19:23.514145 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for
Jan 11 14:19:30 minikube kubelet[4757]: W0111 14:19:30.612595 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for
Jan 11 14:19:31 minikube kubelet[4757]: W0111 14:19:31.754128 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for
The operating system version:
MacOS 10.15.2
The text was updated successfully, but these errors were encountered: