DNS Resolving issues with deploying workload to k3s #4486
Comments
|
I have a few questions. 1 - Where are you trying to resolve hostnames, in the node? Or inside a pod? Are both not working when experiencing the problem? 2 - The problem happens when you deploy the pihole deployment. Does it happen with other deployments too? 3 - Can you share how you are installing the pihole deployment please? |
|
1 - On the node(s). |
I think I did not explain myself correctly, let me clarify :). You mentioned |
|
It used to be any deployment with 1.21.x, but I haven't tested anything else beside Thanks for taking the time to address this issue. |
|
Same problem here. Worked just fine yesterday. Tried older Chart-Versions down to 2.5.1 with no success. |
Also with pihole? |
|
Yes - but drilled it down now. In my case it was a simple dns resolving issue I solved by fixing the nameserver in /etc/resolve.conf. |
|
@manuelbuil Apologies for delay, busy couple of weeks. I've just tried gogs, and downloading images works properly, seems like it's Pihole related. Let me know if you want to investigate this further, if not I'll close the issue and keep investigating on Pihole side. Thank you. |
|
If it's related to Pihole, I'd prefer to close the issue to avoid confusion :) |
|
Will do. I'll post a solution here if I manage to figure it out. |
|
Confirming that indeed it wasn't issue with K3S, but rather with PiHole installation, managed to get it working couple of days ago. If anyone is facing similar issue, feel free to reach out, as this is probably not the right place for that kind of help, and solution is a bit complicated. |
|
@ajvn can you explain your solution? I'm struggling with the same issue. Sorry for commenting it out here, I reckon that this is not the appropriate place, but couldn't find any contact on your github profile |
- When deploying pihole on port 53 of kubernetes cluster the cluster would fail on any type of request, dns lookup. Turns out the VM configured DNS was "127.0.0.53" (a local target), instead of an upstream DNS server like Cloudflare or Google. Refs: k3s-io/k3s#4486 MoJo2600/pihole-kubernetes#88
|
@diogosilva30 Hello, unfortunately I don't recall what the fix was, and my pihole Git history starts 2 days after my comment stating I've found the solution. If you'd like we can continue in the project you referenced in this issue, open an issue there and tag me, maybe we can compare your setup to mine and reverse engineer the differences. P.S. podDnsConfig:
enabled: true
policy: "ClusterFirstWithHostNet"
nameservers:
- 127.0.0.1
- 208.67.222.222 # OpenDNS public nameserver |
Environmental Info:
K3s Version:
Happens with 1.21.4, 1.21.5, and 1.21.6 as well, across RCs, haven't checked other versions.
Node(s) CPU architecture, OS, and Version:
Linux RPI4-2 5.11.0-1021-raspi #22-Ubuntu SMP PREEMPT Wed Oct 6 17:30:38 UTC 2021 aarch64 aarch64 aarch64 GNU/LinuxCluster Configuration:
RPI4s, 8 and 4gb versions.
Output from
/etc/hosts:Describe the bug:
Problem happens while trying to deploy Pihole to the cluster, by being unable to resolve public hostnames of different container image registries.
I've only tried PiHole deployment, but I assume any other would fail with same issue.
Here's more visual way to observe issue across the nodes. This happens when I ping Google before, and during the deployment:
╰─➤ ping google.com PING google.com (142.250.186.174) 56(84) bytes of data. 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=1 ttl=115 time=34.9 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=2 ttl=115 time=28.4 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=3 ttl=115 time=27.5 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=4 ttl=115 time=24.7 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=5 ttl=115 time=24.7 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=6 ttl=115 time=29.0 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=7 ttl=115 time=22.3 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=8 ttl=115 time=22.0 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=9 ttl=115 time=23.0 ms 64 bytes from fra24s08-in-f14.1e100.net (142.250.186.174): icmp_seq=10 ttl=115 time=22.6 ms 64 bytes from 142.250.186.174: icmp_seq=11 ttl=115 time=23.1 ms 64 bytes from 142.250.186.174: icmp_seq=12 ttl=115 time=22.2 ms 64 bytes from 142.250.186.174: icmp_seq=13 ttl=115 time=24.6 ms 64 bytes from 142.250.186.174: icmp_seq=14 ttl=115 time=22.0 ms 64 bytes from 142.250.186.174: icmp_seq=15 ttl=115 time=23.7 ms 64 bytes from 142.250.186.174: icmp_seq=16 ttl=115 time=22.2 ms 64 bytes from 142.250.186.174: icmp_seq=17 ttl=115 time=26.1 ms 64 bytes from 142.250.186.174: icmp_seq=18 ttl=115 time=24.6 ms 64 bytes from 142.250.186.174: icmp_seq=19 ttl=115 time=24.2 ms 64 bytes from 142.250.186.174: icmp_seq=20 ttl=115 time=25.2 ms ^C64 bytes from 142.250.186.174: icmp_seq=21 ttl=115 time=27.5 ms --- google.com ping statistics --- 21 packets transmitted, 21 received, 0% packet loss, time 60193ms rtt min/avg/max/mdev = 22.029/24.981/34.886/3.057 msAfter deployment is removed, it starts working normally again. While it's trying to deploy, DNS will be completely broken across the nodes.
Before deployment/after deployment removal:
telnet registry-1.docker.io 80 Trying 52.204.76.244... Connected to registry-1.docker.io. Escape character is '^]'.Steps To Reproduce:
I'm using Ansible to setup cluster:
Expected behavior:
Being able to deploy workload with working DNS.
Actual behavior:
Not being able to deploy workload because of broken DNS.
Additional context / logs:
Some logs from master k3s:
Backporting
The text was updated successfully, but these errors were encountered: