coredns error: read udp 100.64.0.2:38849->8.8.8.8:53: i/o timeout

[windydayc]

What happen?

I follow https://github.com/sealerio/applications/tree/main/flannel to install k8s with flannel. Although the pods are in the running state, the logs of coredns have errors:

[root@host8c ~]# kubectl get node -owide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
host8c   Ready    master   26m   v1.19.9   10.10.102.155   <none>        CentOS Linux 7 (Core)   3.10.0-693.el7.x86_64   docker://19.3.14
[root@host8c ~]#
[root@host8c ~]#
[root@host8c ~]# kubectl get po -A -owide
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-55bcc669d7-66v6r         1/1     Running   0          7m13s   100.64.0.2      host8c   <none>           <none>
kube-system   coredns-55bcc669d7-pbmmd         1/1     Running   0          7m13s   100.64.0.3      host8c   <none>           <none>
kube-system   etcd-host8c                      1/1     Running   0          7m24s   10.10.102.155   host8c   <none>           <none>
kube-system   kube-apiserver-host8c            1/1     Running   1          7m24s   10.10.102.155   host8c   <none>           <none>
kube-system   kube-controller-manager-host8c   1/1     Running   0          7m24s   10.10.102.155   host8c   <none>           <none>
kube-system   kube-flannel-ds-7sm27            1/1     Running   0          7m14s   10.10.102.155   host8c   <none>           <none>
kube-system   kube-proxy-qx9wk                 1/1     Running   0          7m14s   10.10.102.155   host8c   <none>           <none>
kube-system   kube-scheduler-host8c            1/1     Running   0          7m24s   10.10.102.155   host8c   <none>           <none>
[root@host8c ~]#
[root@host8c ~]#
[root@host8c ~]# kubectl get svc -A
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  36m
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   36m
[root@host8c ~]#
[root@host8c ~]# kubectl logs -n kube-system coredns-55bcc669d7-66v6r
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:38849->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:43750->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:37709->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:50382->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:42229->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:39505->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:48973->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:41398->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:60337->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 334135462374563874.983414830619868657. HINFO: read udp 100.64.0.2:55835->8.8.8.8:53: i/o timeout

Relevant log output?

[root@host8c ~]# cat /etc/resolv.conf
nameserver 8.8.8.8

What you expected to happen?

kubectl logs coredns works well, no errors.

How to reproduce it (as minimally and precisely as possible)?

context is the same as https://github.com/sealerio/applications/tree/main/flannel.

[root@host8c context-k8s-flannel]# sealer build -t k8s1.19.9-flannel -f Kubefile .
Start to Pull Image kubernetes:v1.19.9-alpine
330b931ea81c: pull completed 
0fe0be0c6638: pull completed 
c6bcdcd55fe4: pull completed 
f060a27ccff9: pull completed 
Success to Pull Image kubernetes:v1.19.9-alpine
2022-08-27 13:01:20 [INFO] [executor.go:124] start to check the middleware file
2022-08-27 13:01:20 [INFO] [executor.go:64] run build layer: COPY cni .
2022-08-27 13:01:21 [INFO] [executor.go:64] run build layer: COPY init-kube.sh /scripts/
2022-08-27 13:01:21 [INFO] [executor.go:64] run build layer: COPY kube-flannel.yaml manifests/
2022-08-27 13:01:21 [INFO] [executor.go:96] exec all build instructs success
Pulling image: quay.io/coreos/flannel:v0.14.0
8522d622299c: Download complete 
801bfaa63ef2: Download complete 
e4264a7179f6: Download complete 
bc75ea45ad2e: Download complete 
78648579d12a: Download complete 
3393447261e4: Download complete 
071b96dd834b: Download complete 
4de2f0468a91: Download complete 
Status: images save success
2022-08-27 13:01:21 [INFO] [build.go:100] succeed in building image(k8s1.19.9-flannel) with arch(amd64)


[root@host8c context-k8s-flannel]# sealer run k8s1.19.9-flannel -m 10.10.102.155 -p xxx
2022-08-27 13:01:21 [INFO] [local.go:287] Start to create a new cluster: master [10.10.102.155], worker []
2022-08-27 13:01:21 [INFO] [kube_certs.go:239] APIserver altNames : {map[apiserver.cluster.local:apiserver.cluster.local host8c:host8c kubernetes:kubernetes kubernetes.default:kubernetes.default kubernetes.default.svc:kubernetes.default.svc kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local localhost:localhost] map[10.10.102.155:10.10.102.155 10.103.97.2:10.103.97.2 10.96.0.1:10.96.0.1 127.0.0.1:127.0.0.1 172.16.0.181:172.16.0.181]}
2022-08-27 13:01:21 [INFO] [kube_certs.go:259] Etcd altnames : {map[host8c:host8c localhost:localhost] map[10.10.102.155:10.10.102.155 127.0.0.1:127.0.0.1 ::1:::1]}, commonName : host8c
2022-08-27 13:01:21 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "admin.conf" kubeconfig file

2022-08-27 13:01:21 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "controller-manager.conf" kubeconfig file

2022-08-27 13:01:21 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "scheduler.conf" kubeconfig file

2022-08-27 13:01:21 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "kubelet.conf" kubeconfig file

++ dirname ./init-registry.sh
+ cd .
+ REGISTRY_PORT=5000
+ VOLUME=/var/lib/sealer/data/my-cluster/rootfs/registry
+ REGISTRY_DOMAIN=sea.hub
+ container=sealer-registry
+++ pwd
++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts
+ rootfs=/var/lib/sealer/data/my-cluster/rootfs
+ config=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ htpasswd=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd
+ certs_dir=/var/lib/sealer/data/my-cluster/rootfs/certs
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/images
+ mkdir -p /var/lib/sealer/data/my-cluster/rootfs/registry
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/images/registry.tar ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/images/registry.tar
Loaded image: registry:2.7.1
++ docker ps -aq -f name=sealer-registry
+ '[' '' ']'
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml ']'
+ sed -i s/5000/5000/g /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key     -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd ']'
+ docker run -d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml registry:2.7.1
a4690d79d22ef16e90ca9d00ee1b7e4d0324bbf9b334a5d19903fd34b65b2eea
+ check_registry
+ n=1
+ ((  n <= 3  ))
++ docker inspect --format '{{json .State.Status}}' sealer-registry
+ registry_status='"running"'
+ [[ "running" == \"running\" ]]
+ break
2022-08-27 13:01:21 [INFO] [init.go:259] start to init master0...
2022-08-27 13:01:24 [INFO] [init.go:214] join command is: kubeadm join  apiserver.cluster.local:6443 --token xsc50h.kg6wluufxvnkxe6i \
    --discovery-token-ca-cert-hash sha256:0b9694eaf9e8160e850bbe811bec5865e4fab88724d323c20fc29e2d9f78838e \
    --control-plane --certificate-key fc63c2034d21fa4fbfccdeab5f8b518c48d7f9c5f53acd1a5ad988787722bb83


podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
2022-08-27 13:01:24 [INFO] [local.go:297] Succeeded in creating a new cluster, enjoy it!

Anything else we need to know?

The logs of other pods are normal, no errors.
Change to other k8s versions in kubefile can not solve this problem.

[root@host8c context-k8s-flannel]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=74.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=74.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=74.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=114 time=74.2 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=114 time=73.8 ms
...

[root@host8c ~]# kubectl exec -it busybox sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ #
/ # nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
/ #
/ #
/ # nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
/ #
/ #
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
223 packets transmitted, 0 packets received, 100% packet loss
/ #
/ #
/ # nslookup baidu.com
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'baidu.com'
/ #
/ # nslookup baidu.com.
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'baidu.com.'

[root@host8c ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

What is the version of Sealer you using?

{"gitVersion":"v0.8.6","gitCommit":"884513e","buildDate":"2022-07-12 02:58:54","goVersion":"go1.16.15","compiler":"gc","platform":"linux/amd64"}

What is your OS environment?

centos 7

What is the Kernel version?

Linux host8c 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[windydayc]

I am using a newly created virtual machine without any environment installed beforehand

[windydayc]

This problem occurs when I use sealer to install a k8s cluster, but this problem does not occur when I use the native kubeadm to install k8s cluster.
Using calico image kubernetes:v1.19.8 also does not produce this problem.
So I think there may be some problems with the flannel-based k8s in sealer?

[windydayc]

I tried a lot of different k8s base images and found that some have this problem(such as kubernetes:v1.19.9-alpine) and some don't(such as kubernetes:v1.19.8-alpine, kubernetes:v1.22.8-alpine).

[Stevent-fei]

I tried a lot of different k8s base images and found that some have this problem(such as kubernetes:v1.19.9-alpine) and some don't(such as kubernetes:v1.19.8-alpine, such as kubernetes:v1.22.8-alpine).

The reason for this may be that the mirror has not been updated. We now provide a script to customize the corresponding mirror according to your own needs.
https://github.com/sealerio/basefs/blob/main/auto-build.sh

[windydayc]

I tried a lot of different k8s base images and found that some have this problem(such as kubernetes:v1.19.9-alpine) and some don't(such as kubernetes:v1.19.8-alpine, kubernetes:v1.22.8-alpine).

It's so confusing, I installed the k8s cluster again using kubernetes:v1.22.8-alpine and flannel, and the above problem reappeared.

[windydayc]

I tried a lot of different k8s base images and found that some have this problem(such as kubernetes:v1.19.9-alpine) and some don't(such as kubernetes:v1.19.8-alpine, such as kubernetes:v1.22.8-alpine).

The reason for this may be that the mirror has not been updated. We now provide a script to customize the corresponding mirror according to your own needs. https://github.com/sealerio/basefs/blob/main/auto-build.sh

This script is very slow to install. I've been stuck in the following place for more than an hour, and I don't know if it can be installed successfully.

[root@iZf8z8lt4aao5n3cymq3rzZ ~]# git clone https://github.com/sealerio/basefs.git
[root@iZf8z8lt4aao5n3cymq3rzZ ~]# cd basefs/
[root@iZf8z8lt4aao5n3cymq3rzZ basefs]# ls
auto-build.sh  context  DESIGN.MD  README.md
[root@iZf8z8lt4aao5n3cymq3rzZ basefs]# chmod +x auto-build.sh
[root@iZf8z8lt4aao5n3cymq3rzZ basefs]# ./auto-build.sh --k8s-version=v1.22.8 --cri=docker --platform=amd64
v1.22.8
cri: docker, kubernetes version: v1.22.8, build image name: registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.22.8
+++ echo v1.22.8
+++ grep v
++ kube_install_version=v1.22.8
++ export kube_install_version=v1.22.8
++ kube_install_version=v1.22.8
++ export libseccomp_version=2.5.1
++ libseccomp_version=2.5.1
++ export gperf_version=3.1
++ gperf_version=3.1
++ export conntrack_version=1.4.4
++ conntrack_version=1.4.4
++ export nerdctl_version=0.19.0
++ nerdctl_version=0.19.0
++ export crictl_version=1.24.1
++ crictl_version=1.24.1
++ export containerd_version=1.6.4
++ containerd_version=1.6.4
++ export seautil_version=0.8.5
++ seautil_version=0.8.5
+ ./download.sh docker
download docker version 19.03.14
download registry tarball https://sealer.oss-cn-beijing.aliyuncs.com/auto-build/docker-amd64-registry-image.tar.gz
--2022-09-01 17:58:57--  https://sealer.oss-cn-beijing.aliyuncs.com/auto-build/linux-amd64/conntrack-1.4.4/bin/conntrack
Resolving sealer.oss-cn-beijing.aliyuncs.com (sealer.oss-cn-beijing.aliyuncs.com)... 59.110.185.226
Connecting to sealer.oss-cn-beijing.aliyuncs.com (sealer.oss-cn-beijing.aliyuncs.com)|59.110.185.226|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 71904 (70K) [application/octet-stream]
Saving to: ‘conntrack’

conntrack                               100%[==============================================================================>]  70.22K  --.-KB/s    in 0.08s

2022-09-01 17:58:57 (839 KB/s) - ‘conntrack’ saved [71904/71904]

--2022-09-01 17:58:57--  https://sealer.oss-cn-beijing.aliyuncs.com/auto-build/linux-arm64/conntrack-1.4.4/bin/conntrack
Resolving sealer.oss-cn-beijing.aliyuncs.com (sealer.oss-cn-beijing.aliyuncs.com)... 59.110.185.226
Connecting to sealer.oss-cn-beijing.aliyuncs.com (sealer.oss-cn-beijing.aliyuncs.com)|59.110.185.226|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79592 (78K) [application/octet-stream]
Saving to: ‘conntrack’

conntrack                               100%[==============================================================================>]  77.73K  --.-KB/s    in 0.08s

2022-09-01 17:58:58 (917 KB/s) - ‘conntrack’ saved [79592/79592]

download gperf version 3.1
download libseccomp version 2.5.1
download nerdctl version 0.19.0
nerdctl
containerd-rootless-setuptool.sh
containerd-rootless.sh
//stuck here

[windydayc]

All the above installations are in a clean environment.