Skip to content
This repository has been archived by the owner on Jun 29, 2022. It is now read-only.

Kubelet fails with an "extra space" prefixed in the label value #1280

Open
surajssd opened this issue Dec 23, 2020 · 2 comments · May be fixed by #1291
Open

Kubelet fails with an "extra space" prefixed in the label value #1280

surajssd opened this issue Dec 23, 2020 · 2 comments · May be fixed by #1291
Assignees
@knrt10
Labels
area/kubernetes Core Kubernetes stuff bug Something isn't working

Comments

@surajssd
Copy link
Member

surajssd commented Dec 23, 2020 •
edited
Loading

Description

Self hosted Kubelet fails to start on the worker pools where the key value pair is ill-defined.

Environment and steps to reproduce

  1. Set-up:

Create a cluster on any supported provider with following worker pool config:

  worker_pool "buggy" {
    labels = {
      "foo" = " bar"
    }
  }

Notice the space, at the start, in value of the label.

  1. Task:

Once the cluster is deployed, monitor the state of kubelet pods running on above nodes by running following command:

kubectl -n kube-system get pods -o wide -l k8s-app=kubelet
  1. Error:

And the kubelet on that node is in CrashLoopBackOff state. The reason being there is an extra space in the value field, logs from the kubelet pod:

/usr/local/bin/kubelet --node-ip=10.88.72.135  --anonymous-auth=false --authentication-token-webhook
--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ca.crt --cluster_dns=10.3.0.10
--cluster_domain=cluster.local --cni-conf-dir=/etc/cni/net.d --config=/etc/kubernetes/kubelet.config
--kubeconfig=/var/lib/kubelet/kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig
--rotate-certificates --lock-file=/var/run/lock/kubelet.lock --network-plugin=cni --pod-manifest-path=/etc/kubernetes/manifests
--read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins
--node-labels=foo= bar,node.kubernetes.io/node=metallb.lokomotive.io/my-asn=65000,metallb.lokomotive.io/peer-asn=65530
--register-with-taints=foo=bar:NoSchedule --address=10.88.72.135

[REDACTED]

--volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
F1223 07:04:11.626238   22453 server.go:163] unknown command: bar,node.kubernetes.io/node=
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000124001, 0xc000ada2c0, 0x5b, 0xad)

[REDACTED]

The culprit being the space in flag --node-labels: --node-labels=foo= bar,n....

Expected behavior

We should sanitise the input before passing it along.
@surajssd surajssd added bug Something isn't working area/kubernetes Core Kubernetes stuff labels Dec 23, 2020
@surajssd surajssd changed the title Kubelet fails with extra space in label value Kubelet fails with an "extra space" prefixed in the label value Dec 23, 2020
@knrt10 knrt10 self-assigned this Dec 23, 2020
@invidian
Copy link
Member

I think we can return an error when there is trailing whitespace in label key or value.

knrt10 added a commit that referenced this issue Dec 28, 2020
@knrt10
tinkerbell: fix labels && taints with spaced value
e9fc218 
Now user cannot have spaced key or value in labels, taints map for
worker pools.
Add tests.

closes: #1280
@knrt10 knrt10 linked a pull request Dec 28, 2020 that will close this issue
Open
@knrt10
Copy link
Member

knrt10 commented Dec 28, 2020

Also checking for taints

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@knrt10 knrt10

Labels
area/kubernetes Core Kubernetes stuff bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix labels and taints with spaced value for packet, aws, tinkerbell kinvolk/lokomotive
3 participants
@surajssd @invidian @knrt10