Pods cannot access host.minikube.internal

[Sneaksolid]

What Happened?

When creating a new cluster the pods cannot connect to the host via "host.minikube.internal".

The creation itself works, but after some looking into it i saw the following rule when executing iptables -t nat -L

Chain DOCKER_OUTPUT (2 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             host.minikube.internal  to:127.0.0.11:34415
DNAT       udp  --  anywhere             host.minikube.internal  to:127.0.0.11:39920

which seems like the problem. as the --dport is missing and all traffic gets rerouted instead of only the dns requests.

Thanks in advance for helping.

Docker Version: 20.10.17
Minikube Version: v1.26.0
OS: Arch Linux
Host iptables-nft Version: v1.8.8
Kernel: 5.18.14-arch1-1

Attach the log file

log.txt

Operating System

Other

Driver

Docker

[RA489]

/kind support

[linux019]

I have this bug as well, minikube v1.25.1, kubernetes v1.23.2, Docker 20.10.12 on latest debian with all updates

# minikube start -p my --driver=docker

# kubectl run -i --tty busybox --image=busybox -- sh
# wget www.google.com -O -
Connecting to www.google.com (172.217.16.4:80)
writing to stdout
<!doctype ht......


# wget host.minikube.internal:13456 -O -

Connecting to host.minikube.internal:13456 (192.168.58.1:13456)
wget: error getting response: Resource temporarily unavailable

And with Virtualbox driver I have no any problems with connecting to host

# minikube start -p my-vm --driver=virtualbox
# kubectl run -i --tty busybox --image=busybox -- sh
# wget host.minikube.internal:13456 -O -
Connecting to host.minikube.internal:13456 (192.168.59.1:13456)
writing to stdout
OK...

[klaases]

Hi @Sneaksolid – is this issue still occurring? Were you able to find a solution?

For additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

[linux019]

@klaases the problem is still persist, tested on v1.27.1

[rraval]

I have just encountered this problem and have assembled a one line workaround for now:

$ minikube ssh
docker@minikube:~$ sudo iptables-save | sed -e 's/\(-A DOCKER_OUTPUT .*\)/\1 --dport 53/' | sudo iptables-restore

This needs to be done every time the minikube host restarts (i.e. after every minikube start).

Before:

docker@minikube:~$ sudo iptables -t nat -L DOCKER_OUTPUT
Chain DOCKER_OUTPUT (2 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             host.minikube.internal  to:127.0.0.11:43469
DNAT       udp  --  anywhere             host.minikube.internal  to:127.0.0.11:47246

After:

docker@minikube:~$ sudo iptables -t nat -L DOCKER_OUTPUT
Chain DOCKER_OUTPUT (2 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             host.minikube.internal  tcp dpt:domain to:127.0.0.11:43469
DNAT       udp  --  anywhere             host.minikube.internal  udp dpt:domain to:127.0.0.11:47246
docker@minikube:~$ curl http://host.minikube.internal:8000

Versions:

• minikube: v1.27.0
• docker: Docker version 20.10.18, build v20.10.18
• OS: NixOS on 31d567846255e122846548255101980162bbf641
• Kernel: 5.15.72

---

Tracking this down went down a deep rabbit hole.

I set up a simple HTTP service on the host:

rraval@host:~$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

That HTTP service was inaccessible on the minikube host:

docker@minikube:~$ curl -vv http://host.minikube.internal:8000
*   Trying 192.168.49.1:8000...
* TCP_NODELAY set
* Connected to host.minikube.internal (192.168.49.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: host.minikube.internal:8000
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host host.minikube.internal left intact
curl: (52) Empty reply from server

However pinging the host worked fine:

docker@minikube:~$ ping host.minikube.internal
PING host.minikube.internal (192.168.49.1) 56(84) bytes of data.
64 bytes from host.minikube.internal (192.168.49.1): icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from host.minikube.internal (192.168.49.1): icmp_seq=2 ttl=64 time=0.030 ms
64 bytes from host.minikube.internal (192.168.49.1): icmp_seq=3 ttl=64 time=0.032 ms
^C
--- host.minikube.internal ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 0.030/0.038/0.054/0.010 ms

Wiresharking the bridge network interface revealed that ICMP packets were making it out of the minikube host but the curl invocations were showing no packets. This implies that the problem is on the minikube side (as opposed to some host iptables rule).

After lots of poking, I came across this issue and tried adjusting the iptables rule for DOCKER_OUTPUT to add --dport:

docker@minikube:~$ sudo iptables-save | sed -e 's/\(-A DOCKER_OUTPUT .*\)/\1 --dport 53/' | sudo iptables-restore

docker@minikube:~$ curl -vv http://host.minikube.internal:8000
*   Trying 192.168.49.1:8000...
* TCP_NODELAY set
* Connected to host.minikube.internal (192.168.49.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: host.minikube.internal:8000
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: SimpleHTTP/0.6 Python/3.9.13
< Date: Wed, 26 Oct 2022 18:12:51 GMT
< Content-type: text/html; charset=utf-8
< Content-Length: 3045
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
...

[jsamr]

I could exactly reproduce this issue on a fresh EndeavourOS (Arch derivative) install, and I also reproduced on my Manjaro setup. It's working fine on Ubuntu 22.04. In the three cases, tested with version 1.27.1. @klaases I can see you do functional tests on Ubuntu, could you add an Arch-based workflow?

[djcass44]

I'm also seeing this issue on Fedora 37 w/Minikube v1.28.0. It started happening after upgrading from Fedora 36 (with no changes to Minikube).

The one liner above does solve it for me, however it is temporary.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SumanthAluri-ARAIC]

v1.28.3 problem still persists

[ykidev]

Same on v1.32.0.

[NeroBlackstone]

yes, still not work in 1.33.1