From 5d6da7ec69d8bd252fde49e611d67570d5c82a8b Mon Sep 17 00:00:00 2001 From: Zach Arnold Date: Fri, 21 Sep 2018 10:34:24 -0700 Subject: [PATCH] Revert "Revert "Add CoreDNS details to DNS Debug docs (#10201)"" This reverts commit bb30f4d1fcd6fba2fe6190778ead99f8010033b7. --- .../dns-debugging-resolution.md | 116 ++++++++++++++++-- 1 file changed, 105 insertions(+), 11 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md index 23061cbd74dfd..1a9b7bf7e2289 100644 --- a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md +++ b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md @@ -13,7 +13,7 @@ This page provides hints on diagnosing DNS problems. {{% capture prerequisites %}} * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} * Kubernetes version 1.6 and above. -* The cluster must be configured to use the `kube-dns` addon. +* The cluster must be configured to use the `coredns` (or `kube-dns`) addons. {{% /capture %}} {{% capture steps %}} @@ -68,7 +68,7 @@ nameserver 10.0.0.10 options ndots:5 ``` -Errors such as the following indicate a problem with the kube-dns add-on or +Errors such as the following indicate a problem with the coredns/kube-dns add-on or associated Services: ``` @@ -93,6 +93,17 @@ nslookup: can't resolve 'kubernetes.default' Use the `kubectl get pods` command to verify that the DNS pod is running. +For CoreDNS: +```shell +kubectl get pods --namespace=kube-system -l k8s-app=kube-dns +NAME READY STATUS RESTARTS AGE +... +coredns-7b96bf9f76-5hsxb 1/1 Running 0 1h +coredns-7b96bf9f76-mvmmt 1/1 Running 0 1h +... +``` + +Or for kube-dns: ```shell kubectl get pods --namespace=kube-system -l k8s-app=kube-dns NAME READY STATUS RESTARTS AGE @@ -107,8 +118,26 @@ have to deploy it manually. ### Check for Errors in the DNS pod -Use `kubectl logs` command to see logs for the DNS daemons. +Use `kubectl logs` command to see logs for the DNS containers. +For CoreDNS: +```shell +for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done +``` + +Here is an example of a healthy CoreDNS log: + +``` +.:53 +2018/08/15 14:37:17 [INFO] CoreDNS-1.2.2 +2018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6 +CoreDNS-1.2.2 +linux/amd64, go1.10.3, 2e322f6 +2018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c +``` + + +For kube-dns, there are 3 sets of logs: ```shell kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c kubedns @@ -117,8 +146,8 @@ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c sidecar ``` -See if there is any suspicious log. Letter '`W`', '`E`', '`F`' at the beginning -represent Warning, Error and Failure. Please search for entries that have these +See if there are any suspicious error messages in the logs. In kube-dns, a '`W`', '`E`' or '`F`' at the beginning +of a line represents a Warning, Error or Failure. Please search for entries that have these as the logging level and use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to report unexpected errors. @@ -135,6 +164,8 @@ kube-dns ClusterIP 10.0.0.10 53/UDP,53/TCP 1h ... ``` + +Note that the service name will be "kube-dns" for both CoreDNS and kube-dns deployments. If you have created the service or in the case it should be created by default but it does not appear, see [debugging services](/docs/tasks/debug-application-cluster/debug-service/) for @@ -158,20 +189,83 @@ For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns) in the Kubernetes GitHub repository. + +### Are DNS queries being received/processed? + +You can verify if queries are being received by CoreDNS by adding the `log` plugin to the CoreDNS configuration (aka Corefile). +The CoreDNS Corefile is held in a ConfigMap named `coredns`. To edit it, use the command ... + +``` +kubectl -n kube-system edit configmap coredns +``` + +Then add `log` in the Corefile section per the example below. + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +data: + Corefile: | + .:53 { + log + errors + health + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + upstream + fallthrough in-addr.arpa ip6.arpa + } + prometheus :9153 + proxy . /etc/resolv.conf + cache 30 + loop + reload + loadbalance + } + +``` + +After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods. + +Next, make some queries and view the logs per the sections above in this document. If CoreDNS pods are receiving the queries, you should see them in the logs. + +Here is an example of a query in the log. + +``` +.:53 +2018/08/15 14:37:15 [INFO] CoreDNS-1.2.0 +2018/08/15 14:37:15 [INFO] linux/amd64, go1.10.3, 2e322f6 +CoreDNS-1.2.0 +linux/amd64, go1.10.3, 2e322f6 +2018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f +2018/09/07 15:29:04 [INFO] Reloading complete +172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s + +``` + ## Known issues -Kubernetes installs do not configure the nodes' resolv.conf files to use the -cluster DNS by default, because that process is inherently distro-specific. +Some Linux distributions (e.g. Ubuntu), use a local DNS resolver by default (systemd-resolved). +Systemd-resolved moves and replaces `/etc/resolv.conf` with a stub file that can cause a fatal forwarding +loop when resolving names in upstream servers. This can be fixed manually by using kubelet's `--resolv-conf` flag +to point to the correct `resolv.conf` (With `systemd-resolved`, this is `/run/systemd/resolve/resolv.conf`). +kubeadm 1.11 automatically detects `systemd-resolved`, and adjusts the kubelet flags accordingly. + +Kubernetes installs do not configure the nodes' `resolv.conf` files to use the +cluster DNS by default, because that process is inherently distribution-specific. This should probably be implemented eventually. Linux's libc is impossibly stuck ([see this bug from 2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just -3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to -consume 1 `nameserver` record and 3 `search` records. This means that if a +3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to +consume 1 `nameserver` record and 3 `search` records. This means that if a local installation already uses 3 `nameserver`s or uses more than 3 `search`es, -some of those settings will be lost. As a partial workaround, the node can run +some of those settings will be lost. As a partial workaround, the node can run `dnsmasq` which will provide more `nameserver` entries, but not more `search` -entries. You can also use kubelet's `--resolv-conf` flag. +entries. You can also use kubelet's `--resolv-conf` flag. If you are using Alpine version 3.3 or earlier as your base image, DNS may not work properly owing to a known issue with Alpine.