Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[rke2] pods are duplicated after restart due to the name change #221

Closed
Itxaka opened this issue Jul 28, 2022 · 1 comment · Fixed by rancher/elemental-operator#97 or rancher/elemental-operator#106
Closed

[rke2] pods are duplicated after restart due to the name change #221

Itxaka opened this issue Jul 28, 2022 · 1 comment · Fixed by rancher/elemental-operator#97 or rancher/elemental-operator#106
Labels
area/elemental area/teal kind/bug Something isn't working

Comments

@Itxaka
Copy link
Contributor

Itxaka commented Jul 28, 2022 •
edited
Loading

Operator version: 0.4.1
Elemental version: master (e134aa6)
Rancher version: 2.6.6

Because the first time the rancher-system-agent runs the rke2 deployment the node has the original name () it will create pods based of that name.

But after restart, the node changes name (rancher-NUMBER) so it will try to launch all pods with those names. Some of then can live alongside but others just crash like crazy

rancher-3320:/var/lib/rancher/rke2 # bin/kubectl get pods --all-namespaces
NAMESPACE             NAME                                                                                  READY   STATUS             RESTARTS         AGE
calico-system         calico-kube-controllers-5bbd8d8bdf-mwxr6                                              1/1     Running            1 (14m ago)      19m
calico-system         calico-kube-controllers-5bbd8d8bdf-trrdv                                              1/1     Terminating        0                34m
calico-system         calico-node-2wndp                                                                     1/1     Running            0                34m
calico-system         calico-node-gf8sw                                                                     0/1     CrashLoopBackOff   13 (3m30s ago)   25m
calico-system         calico-typha-58db78ccd5-zx48x                                                         1/1     Running            1 (14m ago)      19m
cattle-fleet-system   fleet-agent-96f6f455c-2nv7x                                                           1/1     Running            1 (14m ago)      19m
cattle-fleet-system   fleet-agent-96f6f455c-2qw5d                                                           1/1     Terminating        0                31m
cattle-system         cattle-cluster-agent-6d74f58464-g6ns5                                                 0/1     CrashLoopBackOff   7 (2m14s ago)    13m
cattle-system         cattle-cluster-agent-d5d7466f-9hmvf                                                   1/1     Terminating        0                32m
kube-system           cloud-controller-manager-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3   1/1     Running            0                35m
kube-system           cloud-controller-manager-rancher-3320                                                 1/1     Running            4 (14m ago)      26m
kube-system           etcd-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3                       1/1     Running            0                35m
kube-system           etcd-rancher-3320                                                                     1/1     Running            1 (14m ago)      26m
kube-system           kube-apiserver-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3             1/1     Running            0                35m
kube-system           kube-apiserver-rancher-3320                                                           1/1     Running            0                13m
kube-system           kube-controller-manager-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3    1/1     Running            0                35m
kube-system           kube-controller-manager-rancher-3320                                                  1/1     Running            4 (14m ago)      26m
kube-system           kube-proxy-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3                 1/1     Running            0                35m
kube-system           kube-proxy-rancher-3320                                                               1/1     Running            0                14m
kube-system           kube-scheduler-m-qemu-standard-pc-q35-ich9-2009-1d566d99-d81a-49ae-b0bb-3             1/1     Running            0                35m
kube-system           kube-scheduler-rancher-3320                                                           1/1     Running            2 (14m ago)      25m
kube-system           rke2-coredns-rke2-coredns-6775f768c8-7v9dv                                            0/1     Pending            0                19m
kube-system           rke2-coredns-rke2-coredns-6775f768c8-rltkf                                            1/1     Running            1 (14m ago)      19m
kube-system           rke2-coredns-rke2-coredns-6775f768c8-swvdp                                            1/1     Terminating        0                34m
kube-system           rke2-coredns-rke2-coredns-autoscaler-7c77dcfb76-sfz4p                                 1/1     Running            1 (14m ago)      19m
kube-system           rke2-coredns-rke2-coredns-autoscaler-7c77dcfb76-tjjrd                                 1/1     Terminating        0                34m
kube-system           rke2-ingress-nginx-controller-5b4fr                                                   1/1     Running            0                32m
kube-system           rke2-ingress-nginx-controller-z6k7g                                                   1/1     Running            1 (14m ago)      25m
kube-system           rke2-metrics-server-8574659c85-2nhzd                                                  1/1     Terminating        0                33m
kube-system           rke2-metrics-server-8574659c85-kbtsk                                                  1/1     Running            1 (14m ago)      19m
tigera-operator       tigera-operator-7b5df56f45-cdmw7                                                      1/1     Running            0                34m
@Itxaka Itxaka mentioned this issue Jul 28, 2022
Merged
@Itxaka Itxaka added this to Elemental Jul 28, 2022
@kkaempf kkaempf moved this to ✅ Done in Elemental Jul 29, 2022
@Itxaka Itxaka moved this from ✅ Done to Archive in Elemental Aug 1, 2022
@Itxaka Itxaka moved this from Archive to ✅ Done in Elemental Aug 1, 2022
mjura added a commit to rancher/elemental-operator that referenced this issue Aug 3, 2022
@mjura
Set fixed hostname and make it persistent
4d8aae9 
For elemental /etc/hostname -> /usr/local/etc/hostname, this file has to be updated
to make change persistent.

Fixes rancher/elemental#221

Signed-off-by: Michal Jura <mjura@suse.com>
@mjura mjura mentioned this issue Aug 3, 2022
Merged
mjura added a commit to rancher/elemental-operator that referenced this issue Aug 3, 2022
@mjura
Set fixed hostname and make it persistent
bc662a5 
For elemental /etc/hostname -> /usr/local/etc/hostname, this file has to be updated
to make change persistent.

Fixes rancher/elemental#221

Signed-off-by: Michal Jura <mjura@suse.com>
mjura added a commit to rancher/elemental-operator that referenced this issue Aug 3, 2022
@mjura
Set fixed hostname and make it persistent
8bcfb18 
For elemental /etc/hostname -> /usr/local/etc/hostname, this file has to be updated
to make change persistent.

Fixes rancher/elemental#221

Signed-off-by: Michal Jura <mjura@suse.com>
@mjura
Copy link
Contributor

mjura commented Aug 3, 2022

There is another changes needed which makes hostname persistent after node restart rancher/elemental-operator#106

mjura added a commit to rancher/elemental-operator that referenced this issue Aug 3, 2022
@mjura
Set fixed hostname and make it persistent
8c9b903 
For elemental /etc/hostname -> /usr/local/etc/hostname, this file has to be updated
to make change persistent.

Fixes rancher/elemental#221

Signed-off-by: Michal Jura <mjura@suse.com>
mjura added a commit to rancher/elemental-operator that referenced this issue Aug 3, 2022
@mjura
Set fixed hostname and make it persistent (#106)
6d730d3 
For elemental /etc/hostname -> /usr/local/etc/hostname, this file has to be updated
to make change persistent.

Fixes rancher/elemental#221

Signed-off-by: Michal Jura <mjura@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/elemental area/teal kind/bug Something isn't working
Projects
Elemental
Archived in project
Milestone
No milestone
2 participants
@Itxaka @mjura