Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

calico version v3.20.0 should be update to v3.23.2 for bug fix #1363

Closed
willzhang opened this issue Jun 29, 2022 · 4 comments
Closed

calico version v3.20.0 should be update to v3.23.2 for bug fix #1363

willzhang opened this issue Jun 29, 2022 · 4 comments
Labels
bug Something isn't working kind/need-to-verify

Comments

@willzhang
Copy link

willzhang commented Jun 29, 2022 •
edited
Loading

What is version of KubeKey has the issue?

version.BuildInfo{Version:"2.2.1", GitCommit:"c056977c", GitTreeState:"", GoVersion:"go1.17.11"}

What is your os environment?

ubuntu 22.04

KubeKey config file

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: node1, address: 192.168.72.40, internalAddress: 192.168.72.40, user: root, password: "123456"}
  - {name: node2, address: 192.168.72.41, internalAddress: 192.168.72.41, user: root, password: "123456"}
  - {name: node3, address: 192.168.72.42, internalAddress: 192.168.72.42, user: root, password: "123456"}
  roleGroups:
    etcd:
    - node1
    control-plane: 
    - node1
    worker:
    - node2
    - node3
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.24.1
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: [https://uyah70su.mirror.aliyuncs.com]
    insecureRegistries: []
  addons: []

A clear and concise description of what happend.

when kubernetes install finished the calico node pods not ready

root@ubuntu:~# kubectl -n kube-system get pods 
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-588c4b86d-p5952   1/1     Running   0          7m2s
calico-node-fdfsx                         0/1     Running   0          7m2s
calico-node-pwlqh                         0/1     Running   0          7m2s
calico-node-rdmjm                         0/1     Running   0          7m2s
coredns-f657fccfd-9s7cq                   1/1     Running   0          7m18s
coredns-f657fccfd-x8224                   1/1     Running   0          7m18s
kube-apiserver-node1                      1/1     Running   0          7m31s
kube-controller-manager-node1             1/1     Running   0          7m30s
kube-proxy-gwpcj                          1/1     Running   0          7m4s
kube-proxy-hcp7w                          1/1     Running   0          7m4s
kube-proxy-j74dc                          1/1     Running   0          7m18s
kube-scheduler-node1                      1/1     Running   0          7m34s
nodelocaldns-46hvs                        1/1     Running   0          7m4s
nodelocaldns-65559                        1/1     Running   0          7m18s
nodelocaldns-pgphz                        1/1     Running   0          7m4s

Relevant log output

root@ubuntu:~# kubectl -n kube-system logs -f calico-node-fdfsx
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-to-host-endpoint" ipVersion=0x4 table="raw"
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-set-endpoint-mark" ipVersion=0x4 table="filter"
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 580: Chain became referenced, marking it for programming chainName="cali-sm-cali76675a41a3d"
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-from-endpoint-mark" ipVersion=0x4 table="filter"
2022-06-29 14:56:03.256 [INFO][15095] felix/endpoint_mgr.go 476: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"kube-system/calico-kube-controllers-588c4b86d-p5952", EndpointId:"eth0"}
2022-06-29 14:56:03.256 [INFO][15095] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"kube-system/calico-kube-controllers-588c4b86d-p5952", EndpointId:"eth0"}
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-fip-dnat" ipVersion=0x4 table="nat"
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-fip-snat" ipVersion=0x4 table="nat"
2022-06-29 14:56:03.256 [INFO][15095] felix/masq_mgr.go 144: IPAM pools updated, refreshing iptables rule ipVersion=0x4
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-nat-outgoing" ipVersion=0x4 table="nat"
2022-06-29 14:56:03.256 [INFO][15095] felix/ipip_mgr.go 222: All-hosts IP set out-of sync, refreshing it.
2022-06-29 14:56:03.256 [INFO][15095] felix/ipsets.go 126: Queueing IP set for creation family="inet" setID="all-hosts-net" setType="hash:net"
2022-06-29 14:56:03.256 [INFO][15095] felix/table.go 506: Queueing update of chain. chainName="cali-cidr-block" ipVersion=0x4 table="filter"
2022-06-29 14:56:03.339 [INFO][15095] felix/wireguard.go 1616: Trying to connect to linkClient
2022-06-29 14:56:03.339 [INFO][15095] felix/route_table.go 417: Trying to connect to netlink
2022-06-29 14:56:03.340 [INFO][15095] felix/route_table.go 417: Trying to connect to netlink
2022-06-29 14:56:03.343 [INFO][15095] felix/route_rule.go 176: Trying to connect to netlink
2022-06-29 14:56:03.345 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.345 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.345 [INFO][15095] felix/wireguard.go 570: Public key out of sync or updated ourPublicKey=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
2022-06-29 14:56:03.349 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.353 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.353 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.355 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.357 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.357 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.362 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.364 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.364 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.372 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.374 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.374 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.390 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.392 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.392 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.400 [INFO][15095] felix/int_dataplane.go 1060: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calico_tmp_B"
2022-06-29 14:56:03.401 [INFO][15095] felix/int_dataplane.go 1060: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calico_tmp_A"
2022-06-29 14:56:03.406 [INFO][15095] felix/int_dataplane.go 1060: Linux interface addrs changed. addrs=<nil> ifaceName="calico_tmp_B"
2022-06-29 14:56:03.406 [INFO][15095] felix/int_dataplane.go 1060: Linux interface addrs changed. addrs=<nil> ifaceName="calico_tmp_A"
2022-06-29 14:56:03.425 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.427 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.427 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.491 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.493 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.493 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.621 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.623 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.623 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
2022-06-29 14:56:03.880 [INFO][15095] felix/ipsets.go 308: Retrying after an ipsets update failure... family="inet"
2022-06-29 14:56:03.881 [ERROR][15095] felix/ipsets.go 570: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.1: Kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.\n"
2022-06-29 14:56:03.882 [WARNING][15095] felix/ipsets.go 318: Failed to resync with dataplane error=exit status 1 family="inet"
^C

Additional information

Upstream issue: projectcalico/calico#5717
@willzhang willzhang added the bug Something isn't working label Jun 29, 2022
@willzhang willzhang changed the title calico version v3.20.0 should be update for bug fix calico version v3.20.0 should be update to v2.22.4 for bug fix Jun 29, 2022
@willzhang willzhang changed the title calico version v3.20.0 should be update to v2.22.4 for bug fix calico version v3.20.0 should be update to v3.23.2 for bug fix Jun 29, 2022
@pixiake pixiake mentioned this issue Jun 30, 2022
Closed
@24sama
Copy link
Collaborator

24sama commented Jul 1, 2022

fixed by: #1366

/kind need-to-verify

@fuergaosi233
Copy link

I have the same problem, how can I fix it by modifying the config?

@24sama
Copy link
Collaborator

24sama commented Jul 20, 2022

I have the same problem, how can I fix it by modifying the config?

Please see: #1282 (comment)

@24sama
Copy link
Collaborator

24sama commented Jul 26, 2022

Hi, the kk v2.2.2 has been released. And this version has already upgraded the calico version v3.20.0 to v3.23.2 (#1366). And I guess this issue can be closed.

@24sama 24sama closed this as completed Jul 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working kind/need-to-verify
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fuergaosi233 @willzhang @24sama @ks-ci-bot