Pods without internet connection

[mutech]

Pods have no access to local network or internet.

That occurs after installing Ubuntu 20.04 (server or desktop - same) and installing microk8s (selecting it in server installation or right after desktop installation finishes).

On an existing installation on Ubuntu 20.04 running in a VMWare VM, the same procedure works after running sudo iptables -P FORWARD ACCEPT.

The failing installation is on a repurposed iMac 2010. Since it works in the VM, that's probably contributing to the problem, even though I don't really understand why, since networking otherwise works without issues (cable connection, no changing IP addresses)

macws01_inspection-report-20201021_180959.tar.gz

@ktsakalozos (from #1484)

[ktsakalozos]

Hi @mutech

From the attached tarball I do not see what might be wrong. I see the two CNI related pods up and running but nothing else.

Could you reproduce the issue in a generic way (maybe with [1]) and then collect the logs with the microk8s inspect? Also please describe the steps you have taken to hit the problem.

[1] https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

[mutech]

Hi @ktsakalozos & thanks for taking the time to look into this!

Meanwhile I fiddled around with that installation (installing/uninstalling docker for another purpose which is still seen after apt remove docker.io --purger by microk8s inspect). I also installed dns, ingress, storage and registry. I'm not sure if some of these changes have an impact on this issue, I'll do a fresh install later.

I added a pod like this:

apiVersion: v1
kind: Pod
metadata:
        name: netshoot
        namespace: default
spec:
        containers:
        - name: netshoot
          image: nicolaka/netshoot
          command:
            - sleep
            - "3600"
          imagePullPolicy: IfNotPresent
        restartPolicy: Always

• DNS works
• ping 8.8.8.8 (or anything else really) works too
• telnet internethost 25 / 22 both work

Everything that's using http and https fails with timeouts, f.e. apk add openssh

Here is the inspect report:
inspection-report-20201022_195327.tar.gz

So it very much looks like there is an http proxy involved or some protocol specific firewall setup. However I did not consciously setup any nor did I find anything related to it.

My original problem with microk8s was to get Eclipse che to work. I initially failed on Ubuntu desktop (where forwarding is not enabled by default and DNS resolution fails first followed by this problem once that was fixed). Here is the related ticket: eclipse-che/che#18128 (comment)

(I closed that ticket because I thought fixing the firewall (forward accept) setting solves the problem, but discovered later that http traffic is still not working)

[mutech]

I found the problem ("DNS works" above was a bit superficial):

Running host -v inside the pod:

$ host -v google.com
Trying "www.google.com.default.svc.cluster.local"
Trying "www.google.com.svc.cluster.local"
Trying "www.google.com.cluster.local"
Trying "www.google.com.MY.LOCAL.DOMAIN"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21087
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.MY.LOCAL.DOMAIN.	IN	A
... (rest omitted)

The NS for MY.LOCAL.DOMAIN has a wildcard record, so instead of resolving www.google.com, it resolves to the wildcard IP.

Is there a way to keep k8s from using the host domain like this?