Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: kubelet can't register static node to the k8s cluster #1554

Open
JKBGIT1 opened this issue Oct 24, 2024 · 0 comments
Open

Bug: kubelet can't register static node to the k8s cluster #1554

JKBGIT1 opened this issue Oct 24, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@JKBGIT1
Copy link
Contributor

JKBGIT1 commented Oct 24, 2024

Current Behaviour

Kubelet services on static nodes failed to register these nodes to the k8s cluster when applying the InputManifest from Steps To Reproduce part. This resulted in the following error in the kuber.

2024-10-23T15:17:02Z WRN Node eu01-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node eu02-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node eu03-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node db-eu01-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node db-eu02-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node db-eu03-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node db-eu04-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:02Z WRN Node db-eu05-01 was not found in cluster [auto-eu-ubcqj4n-01 auto-us-1k0pirq-01 auto-us-1k0pirq-02 control01-vag48zt-01 control01-vag48zt-02 control01-vag48zt-03 db-sin01-ywmnlfl-01 db-us01-w0dv6fw-01 db-us02-i1us555-01 ] cluster=wox01-cluster-s866pid module=kuber
wox01-cluster-s866pid   storageclass.storage.k8s.io/longhorn-static-provider-zone created
2024-10-23T15:17:04Z INF Longhorn successfully set up cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:04Z INF Storing kubeconfig cluster=wox01-cluster-s866pid module=kuber
wox01-cluster-s866pid-kubeconfig        secret/wox01-cluster-s866pid-kubeconfig created
2024-10-23T15:17:04Z INF Kubeconfig was successfully stored cluster=wox01-cluster-s866pid module=kuber
2024-10-23T15:17:04Z INF Storing cluster metadata cluster=wox01-cluster-s866pid module=kuber
wox01-cluster-s866pid-metadata  secret/wox01-cluster-s866pid-metadata created
2024-10-23T15:17:05Z INF Cluster metadata was successfully stored cluster=wox01-cluster-s866pid module=kuber
wox01-cluster-s866pid   node/control01-vag48zt-01 patched
wox01-cluster-s866pid   node/control01-vag48zt-02 patched
wox01-cluster-s866pid   node/control01-vag48zt-03 patched
wox01-cluster-s866pid   node/auto-eu-ubcqj4n-01 patched
wox01-cluster-s866pid   node/auto-us-1k0pirq-01 patched
wox01-cluster-s866pid   node/auto-us-1k0pirq-02 patched
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu01-01" not found
2024-10-23T15:17:09Z INF Next retry in 10s... module=kuber
2024-10-23T15:17:19Z WRN Retrying command kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu01-01" not found
2024-10-23T15:17:20Z WRN Error encountered while executing kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f : exit status 1 module=kuber
2024-10-23T15:17:20Z INF Next retry in 20s... module=kuber
2024-10-23T15:17:40Z WRN Retrying command kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu01-01" not found
2024-10-23T15:17:40Z WRN Error encountered while executing kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f : exit status 1 module=kuber
2024-10-23T15:17:40Z INF Next retry in 40s... module=kuber
2024-10-23T15:18:20Z WRN Retrying command kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu01-01" not found
2024-10-23T15:18:21Z WRN Error encountered while executing kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f : exit status 1 module=kuber
2024-10-23T15:18:21Z ERR Command kubectl patch node eu01-01 -p '{"spec":{"providerID":"claudie://eu01-01"}}' --kubeconfig /tmp/k8b369c675fb444f89bf5023775fc6f8f was not successful after 3 retries module=kuber
2024-10-23T15:18:21Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://eu01-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=eu01-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu02-01" not found
2024-10-23T15:18:21Z INF Next retry in 10s... module=kuber
2024-10-23T15:18:31Z WRN Retrying command kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu02-01" not found
2024-10-23T15:18:32Z WRN Error encountered while executing kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635 : exit status 1 module=kuber
2024-10-23T15:18:32Z INF Next retry in 20s... module=kuber
2024-10-23T15:18:52Z WRN Retrying command kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu02-01" not found
2024-10-23T15:18:52Z WRN Error encountered while executing kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635 : exit status 1 module=kuber
2024-10-23T15:18:52Z INF Next retry in 40s... module=kuber
2024-10-23T15:19:32Z WRN Retrying command kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu02-01" not found
2024-10-23T15:19:33Z WRN Error encountered while executing kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635 : exit status 1 module=kuber
2024-10-23T15:19:33Z ERR Command kubectl patch node eu02-01 -p '{"spec":{"providerID":"claudie://eu02-01"}}' --kubeconfig /tmp/k7fe2d193cb77471ca86ae36ad5ab7635 was not successful after 3 retries module=kuber
2024-10-23T15:19:33Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://eu02-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=eu02-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu03-01" not found
2024-10-23T15:19:33Z INF Next retry in 10s... module=kuber
2024-10-23T15:19:43Z WRN Retrying command kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu03-01" not found
2024-10-23T15:19:44Z WRN Error encountered while executing kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1 : exit status 1 module=kuber
2024-10-23T15:19:44Z INF Next retry in 20s... module=kuber
2024-10-23T15:20:04Z WRN Retrying command kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu03-01" not found
2024-10-23T15:20:05Z WRN Error encountered while executing kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1 : exit status 1 module=kuber
2024-10-23T15:20:05Z INF Next retry in 40s... module=kuber
2024-10-23T15:20:45Z WRN Retrying command kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "eu03-01" not found
2024-10-23T15:20:45Z WRN Error encountered while executing kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1 : exit status 1 module=kuber
2024-10-23T15:20:45Z ERR Command kubectl patch node eu03-01 -p '{"spec":{"providerID":"claudie://eu03-01"}}' --kubeconfig /tmp/kf3185d443b15465fac548980b8e1aba1 was not successful after 3 retries module=kuber
2024-10-23T15:20:45Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://eu03-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=eu03-01
wox01-cluster-s866pid   node/db-sin01-ywmnlfl-01 patched
wox01-cluster-s866pid   node/db-us01-w0dv6fw-01 patched
wox01-cluster-s866pid   node/db-us02-i1us555-01 patched
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu01-01" not found
2024-10-23T15:20:47Z INF Next retry in 10s... module=kuber
2024-10-23T15:20:57Z WRN Retrying command kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu01-01" not found
2024-10-23T15:20:58Z WRN Error encountered while executing kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45 : exit status 1 module=kuber
2024-10-23T15:20:58Z INF Next retry in 20s... module=kuber
2024-10-23T15:21:18Z WRN Retrying command kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu01-01" not found
2024-10-23T15:21:18Z WRN Error encountered while executing kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45 : exit status 1 module=kuber
2024-10-23T15:21:18Z INF Next retry in 40s... module=kuber
2024-10-23T15:21:58Z WRN Retrying command kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu01-01" not found
2024-10-23T15:21:59Z WRN Error encountered while executing kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45 : exit status 1 module=kuber
2024-10-23T15:21:59Z ERR Command kubectl patch node db-eu01-01 -p '{"spec":{"providerID":"claudie://db-eu01-01"}}' --kubeconfig /tmp/k4933fc4a8be84eb0a6739cfd0bf2bb45 was not successful after 3 retries module=kuber
2024-10-23T15:21:59Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://db-eu01-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=db-eu01-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu02-01" not found
2024-10-23T15:22:00Z INF Next retry in 10s... module=kuber
2024-10-23T15:22:10Z WRN Retrying command kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu02-01" not found
2024-10-23T15:22:10Z WRN Error encountered while executing kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9 : exit status 1 module=kuber
2024-10-23T15:22:10Z INF Next retry in 20s... module=kuber
2024-10-23T15:22:30Z WRN Retrying command kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu02-01" not found
2024-10-23T15:22:31Z WRN Error encountered while executing kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9 : exit status 1 module=kuber
2024-10-23T15:22:31Z INF Next retry in 40s... module=kuber
2024-10-23T15:23:11Z WRN Retrying command kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu02-01" not found
2024-10-23T15:23:11Z WRN Error encountered while executing kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9 : exit status 1 module=kuber
2024-10-23T15:23:11Z ERR Command kubectl patch node db-eu02-01 -p '{"spec":{"providerID":"claudie://db-eu02-01"}}' --kubeconfig /tmp/k2e182322bba24c66ab658c9e01bd5db9 was not successful after 3 retries module=kuber
2024-10-23T15:23:11Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://db-eu02-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=db-eu02-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu03-01" not found
2024-10-23T15:23:12Z INF Next retry in 10s... module=kuber
2024-10-23T15:23:22Z WRN Retrying command kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu03-01" not found
2024-10-23T15:23:22Z WRN Error encountered while executing kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e : exit status 1 module=kuber
2024-10-23T15:23:22Z INF Next retry in 20s... module=kuber
2024-10-23T15:23:42Z WRN Retrying command kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu03-01" not found
2024-10-23T15:23:43Z WRN Error encountered while executing kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e : exit status 1 module=kuber
2024-10-23T15:23:43Z INF Next retry in 40s... module=kuber
2024-10-23T15:24:23Z WRN Retrying command kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu03-01" not found
2024-10-23T15:24:23Z WRN Error encountered while executing kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e : exit status 1 module=kuber
2024-10-23T15:24:23Z ERR Command kubectl patch node db-eu03-01 -p '{"spec":{"providerID":"claudie://db-eu03-01"}}' --kubeconfig /tmp/kd8e3da9fc163428bae7848085511568e was not successful after 3 retries module=kuber
2024-10-23T15:24:23Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://db-eu03-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=db-eu03-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu04-01" not found
2024-10-23T15:24:24Z INF Next retry in 10s... module=kuber
2024-10-23T15:24:34Z WRN Retrying command kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu04-01" not found
2024-10-23T15:24:34Z WRN Error encountered while executing kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682 : exit status 1 module=kuber
2024-10-23T15:24:34Z INF Next retry in 20s... module=kuber
2024-10-23T15:24:54Z WRN Retrying command kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu04-01" not found
2024-10-23T15:24:55Z WRN Error encountered while executing kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682 : exit status 1 module=kuber
2024-10-23T15:24:55Z INF Next retry in 40s... module=kuber
2024-10-23T15:25:35Z WRN Retrying command kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu04-01" not found
2024-10-23T15:25:36Z WRN Error encountered while executing kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682 : exit status 1 module=kuber
2024-10-23T15:25:36Z ERR Command kubectl patch node db-eu04-01 -p '{"spec":{"providerID":"claudie://db-eu04-01"}}' --kubeconfig /tmp/k410d6b81a6284772998bf2ce13cbb682 was not successful after 3 retries module=kuber
2024-10-23T15:25:36Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://db-eu04-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=db-eu04-01
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu05-01" not found
2024-10-23T15:25:36Z INF Next retry in 10s... module=kuber
2024-10-23T15:25:46Z WRN Retrying command kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222... (1/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu05-01" not found
2024-10-23T15:25:47Z WRN Error encountered while executing kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222 : exit status 1 module=kuber
2024-10-23T15:25:47Z INF Next retry in 20s... module=kuber
2024-10-23T15:26:07Z WRN Retrying command kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222... (2/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu05-01" not found
2024-10-23T15:26:07Z WRN Error encountered while executing kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222 : exit status 1 module=kuber
2024-10-23T15:26:07Z INF Next retry in 40s... module=kuber
2024-10-23T15:26:47Z WRN Retrying command kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222... (3/3) module=kuber
wox01-cluster-s866pid   Error from server (NotFound): nodes "db-eu05-01" not found
2024-10-23T15:26:48Z WRN Error encountered while executing kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222 : exit status 1 module=kuber
2024-10-23T15:26:48Z ERR Command kubectl patch node db-eu05-01 -p '{"spec":{"providerID":"claudie://db-eu05-01"}}' --kubeconfig /tmp/kab619570289645f694c257dab42f3222 was not successful after 3 retries module=kuber
2024-10-23T15:26:48Z ERR Error while patching node with patch {"spec":{"providerID":"claudie://db-eu05-01"}} error="exit status 1" cluster=wox01-cluster-s866pid module=kuber node=db-eu05-01
2024-10-23T15:26:48Z ERR Error while patching node provider ID error="error while patching one or more nodes with providerID" cluster=wox01-cluster-s866pid module=kuber

In the output of kubectl get nodes you can see the static nodes are missing although they had running and healthy kubelet service.

NAME                   STATUS   ROLES           AGE   VERSION
auto-eu-ubcqj4n-01     Ready    <none>          16h   v1.30.5
auto-us-1k0pirq-01     Ready    <none>          16h   v1.30.5
auto-us-1k0pirq-02     Ready    <none>          16h   v1.30.5
control01-vag48zt-01   Ready    control-plane   16h   v1.30.5
control01-vag48zt-02   Ready    control-plane   16h   v1.30.5
control01-vag48zt-03   Ready    control-plane   16h   v1.30.5
db-sin01-ywmnlfl-01    Ready    <none>          16h   v1.30.5
db-us01-w0dv6fw-01     Ready    <none>          16h   v1.30.5
db-us02-i1us555-01     Ready    <none>          16h   v1.30.5

The kube-eleven logs which are related to static nodes. It's worth mentioning that kube-eleven finished its job successfully.

wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"db-eu01-01\", assuming it was garbage-collected" node=65.21.206.124
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"eu03-01\", assuming it was garbage-collected" node=95.216.224.11
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"db-eu05-01\", assuming it was garbage-collected" node=65.21.205.253
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"db-eu04-01\", assuming it was garbage-collected" node=95.216.245.56
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"db-eu03-01\", assuming it was garbage-collected" node=65.21.206.83
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"eu01-01\", assuming it was garbage-collected" node=95.216.97.169
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"eu02-01\", assuming it was garbage-collected" node=95.216.64.35
wox01-cluster-s866pid   time="15:15:16 UTC" level=info msg="No CSR found for node \"db-eu02-01\", assuming it was garbage-collected" node=37.27.112.183
...
wox01-cluster-s866pid   time="15:15:49 UTC" level=info msg="Approve pending CSR \"csr-c2sd4\" for username \"system:node:auto-us-1k0pirq-01\"" node=5.78.126.93
wox01-cluster-s866pid   time="15:15:50 UTC" level=info msg="Approve pending CSR \"csr-62x4f\" for username \"system:node:db-us01-w0dv6fw-01\"" node=5.78.64.255
wox01-cluster-s866pid   time="15:15:50 UTC" level=info msg="Approve pending CSR \"csr-cjb9g\" for username \"system:node:auto-us-1k0pirq-02\"" node=5.78.108.149
wox01-cluster-s866pid   time="15:15:50 UTC" level=info msg="Approve pending CSR \"csr-5ftw8\" for username \"system:node:db-sin01-ywmnlfl-01\"" node=5.223.50.70
wox01-cluster-s866pid   time="15:15:50 UTC" level=info msg="Approve pending CSR \"csr-76dlx\" for username \"system:node:db-us02-i1us555-01\"" node=5.78.130.23
wox01-cluster-s866pid   time="15:15:50 UTC" level=info msg="Approve pending CSR \"csr-pst64\" for username \"system:node:auto-eu-ubcqj4n-01\"" node=95.217.154.127

The kubelet service on each static nodes restarted ~20 times. Each time with the following error

Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 22.
Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base (kubelet)[10466]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS
Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base kubelet[10466]: E1023 16:08:11.918543   10466 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error:  failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Oct 23 16:08:11 Ubuntu-2404-noble-amd64-base systemd[1]: kubelet.service: Failed with result 'exit-code'.

Kubelet logged the following error when failed to register the static node.

kubelet_node_status.go:96] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope" node="eu01-01"

Expected Behaviour

Kubelet doesn't fail to register the static node.

Steps To Reproduce

Apply the following InputManifest and wait until it fails in kuber.

apiVersion: claudie.io/v1beta1
kind: InputManifest
metadata:
  name: wox
  labels:
    app.kubernetes.io/part-of: claudie
    inputmanifest.version: v1
spec:
  providers:
    # Cloudflare DNS provider.
    - name: cloudflare
      providerType: cloudflare
      secretRef:
        name: cloudflare-secret
        namespace: wox

    # Hetzner Cloud provider.
    - name: hetzner
      providerType: hetzner
      secretRef:
        name: hetzner-secret
        namespace: wox

  nodePools:
    dynamic:
      - name: control01
        providerSpec:
          name: hetzner
          region: hel1
          zone: hel1-dc2
        count: 3
        serverType: cpx21
        image: ubuntu-24.04
#
#      - name: control02
#        providerSpec:
#          name: hetzner
#          region: nbg1
#          zone: nbg1-dc3
#        count: 1
#        serverType: cpx21
#        image: ubuntu-24.04
#
#      - name: control03
#        providerSpec:
#          name: hetzner
#          region: fsn1
#          zone: fsn1-dc14
#        count: 1
#        serverType: cx32
#        image: ubuntu-24.04

      - name: auto-eu
        providerSpec:
          name: hetzner
          region: hel1
          zone: hel1-dc2
        serverType: ccx23
        storageDiskSize: 0
        image: ubuntu-24.04
        autoscaler:
          min: 1
          max: 20
        labels:
          region: europe
          type: app
        annotations:
          node.longhorn.io/default-node-tags: '["eu"]'

      - name: auto-us
        providerSpec:
          name: hetzner
          region: hil
          zone: hil-dc1
        serverType: ccx23
        storageDiskSize: 0
        image: ubuntu-24.04
        autoscaler:
          min: 2
          max: 20
        labels:
          region: us
          type: app
        annotations:
          node.longhorn.io/default-node-tags: '["us"]'

      - name: lb-eu
        providerSpec:
          name: hetzner
          region: hel1
          zone: hel1-dc2
        count: 1
        serverType: cpx21
        image: ubuntu-24.04

      - name: lb-sin
        providerSpec:
          name: hetzner
          region: sin
          zone: sin-dc1
        count: 1
        serverType: cpx11
        image: ubuntu-24.04

      - name: lb-us
        providerSpec:
          name: hetzner
          region: hil
          zone: hil-dc1
        count: 1
        serverType: ccx13
        image: ubuntu-24.04

      - name: db-sin01
        providerSpec:
          name: hetzner
          region: sin
          zone: sin-dc1
        count: 1
        storageDiskSize: 0
        serverType: ccx33
        image: ubuntu-24.04
        labels:
          region: sin
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-sin"]'

      - name: db-us01
        providerSpec:
          name: hetzner
          region: hil
          zone: hil-dc1
        count: 1
        storageDiskSize: 0
        serverType: ccx33
        image: ubuntu-24.04
        labels:
          region: us
          zone: us02
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-us"]'

      - name: db-us02
        providerSpec:
          name: hetzner
          region: hil
          zone: hil-dc1
        count: 1
        storageDiskSize: 0
        serverType: ccx33
        image: ubuntu-24.04
        labels:
          region: us
          zone: us03
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-us"]'

    static:
      - name: eu01
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: app
        annotations:
          node.longhorn.io/default-node-tags: '["eu"]'
      - name: eu02
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: app
        annotations:
          node.longhorn.io/default-node-tags: '["eu"]'
      - name: eu03
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: app
        annotations:
          node.longhorn.io/default-node-tags: '["eu"]'

      # Databases
      - name: db-eu01
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-eu"]'
      - name: db-eu02
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-eu"]'
      - name: db-eu03
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-eu"]'
      - name: db-eu04
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-eu"]'
      - name: db-eu05
        nodes:
          - endpoint: <public-IP>
            secretRef:
              name: ssh-key
              namespace: wox
        labels:
          region: europe
          type: db
        annotations:
          node.longhorn.io/default-node-tags: '["db-eu"]'

  kubernetes:
    clusters:
      - name: wox01-cluster
        version: v1.30.5
        network: 192.168.2.0/24
        pools:
          control:
            - control01
          compute:
            - auto-eu
            - auto-us
            - eu01
            - eu02
            - eu03

            - db-sin01
            - db-us01
            - db-us02
            - db-eu01
            - db-eu02
            - db-eu03
            - db-eu04
            - db-eu05

  loadBalancers:
    roles:
      - name: apiserver
        protocol: tcp
        port: 6443
        targetPort: 6443
        targetPools:
          - control01
      - name: eu-https
        protocol: tcp
        port: 443
        targetPort: 32008
        targetPools:
          - auto-eu
          - eu01
          - eu02
          - eu03
      - name: eu-http
        protocol: tcp
        port: 80
        targetPort: 30645
        targetPools:
          - auto-eu
          - eu01
          - eu02
          - eu03
      - name: us-https
        protocol: tcp
        port: 443
        targetPort: 32008
        targetPools:
          - auto-us
      - name: us-http
        protocol: tcp
        port: 80
        targetPort: 30645
        targetPools:
          - auto-us
      - name: sin-https
        protocol: tcp
        port: 443
        targetPort: 32008
        targetPools:
          - db-sin01
      - name: sin-http
        protocol: tcp
        port: 80
        targetPort: 30645
        targetPools:
          - db-sin01
    clusters:
      - name: lb-eu
        roles:
          - apiserver
          - eu-https
          - eu-http
        dns:
          dnsZone: <dns-zone>
          provider: cloudflare
          hostname: loadbalancer
        targetedK8s: wox01-cluster
        pools:
          - lb-eu
      - name: lb-sin
        roles:
          - sin-https
          - sin-http
        dns:
          dnsZone: <dns-zone>
          provider: cloudflare
          hostname: loadbalancer-sin
        targetedK8s: wox01-cluster
        pools:
          - lb-sin
      - name: lb-us
        roles:
          - us-https
          - us-http
        dns:
          dnsZone: <dns-zone>
          provider: cloudflare
          hostname: loadbalancer-us
        targetedK8s: wox01-cluster
        pools:
          - lb-us
@JKBGIT1 JKBGIT1 added the bug Something isn't working label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JKBGIT1