-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1.7.1 with hostdns and forwardKubeDNSToHost doesn't resolve anything #8698
Comments
Greetings! Can you provide |
sure! with
i get ~13k lines, here's the last few:
with
I get
As soon as the patch is applied and coredns restarted, I start immediately seeing issues, for example in my homeassistant logs:
and from the coredns deployment logs itself:
changing
|
Same issue for me, but unfortunately disabling hostDNS features doesn't resolve the issue. I am using my own DNS servers, however using public DNS servers didn't help. It worked fine using version 1.6.7, failed to work from 1.7.0, keeps failing in 1.7.1. |
Let's not mix different issues in one ticket please. |
@evanrich what is the CNI you're using? |
Cilium v1.15.4 |
I'm seeing the same problem on Talos 1.7.1 (also upgraded from earlier versions), Kubernetes 1.29.1, Cilium 1.15.4. I am using DHCP-discovered public DNS servers run by Hetzner. Hubble (Cilium packet inspection) reports that the UDP requests from CoreDNS to the Talos DNS service IP (10.96.0.9 in my case) are delivered, but the response packets from 10.96.0.9 to CoreDNS pod are dropped with the reason |
I have the same error. I'm using talos v1.7.1 and cilium v1.14.7 |
Check if you are using bpf.masquerade if yes and you did not specify CIDRs manually, then with common private CIDRs you will get above error. Try to set bpf.masquerade option to false and check if that works. |
Sounds very plausible. However, bpf masquerade is disabled for my use case, but I can see that iptables masquerade for ipv4 is enabled. I would assume disabling this would have the same effect? Edit:
It seems to me that masquerading is a very likely culprit, but I'm not sure how exactly yet. Will keep digging. |
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
…or pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host DNS resolver as upstream for Kubernetes CoreDNS pods. Credits go to Julian Wiedmann. For siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
…or pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host DNS resolver as upstream for Kubernetes CoreDNS pods. Credits go to Julian Wiedmann. For siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
…or pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host DNS resolver as upstream for Kubernetes CoreDNS pods. Credits go to Julian Wiedmann. For siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
…or pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host DNS resolver as upstream for Kubernetes CoreDNS pods. Credits go to Julian Wiedmann. For siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
…or pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host DNS resolver as upstream for Kubernetes CoreDNS pods. Credits go to Julian Wiedmann. For siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
The fix is coming, thanks for reporting it, it's indeed the TTL. It's only related to |
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
Reopened until there is 1.7 backport. |
This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes siderolabs#8698. Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com> (cherry picked from commit 53f5489)
Closed per #8758 |
@DmitriyMV not sure if this is related, but after upgrading 1.7.1->1.7.2, while better, I now see other errors:
this is based off the following config:
The only thing i flipped from 1.7.1. to 1.7.2 was the forward to host. |
1.7.3 fixes the errors above |
Bug Report
Description
This is on a cluster that has been upgraded (1.6.x->1.7.x), not fresh
after applying the following patch:
nothing seems to resolve dns, either in the cluster or externally
getupstream:
resolv.conf
resolvers
CoreDNS was restarted twice after applying the patch.
Logs
Environment
Reverting the patch (false/false) fixes dns again.
FWIW, here's my coredns configmap:
The text was updated successfully, but these errors were encountered: