Karpenter ignores extra labels coming from kubelet-extra-args in launchtemplate

[yerenkow]

Version

Karpenter: v0.8.2
Kubernetes: v1.21

Expected Behavior

Node should set labels coming from extra kubelet args (in launch template user data) early, so daemonsets should see it.

Actual Behavior

It doesn't work, as aws-node pod doesn't see label.

Steps to Reproduce the Problem

We use launch template with certain customization done in the user-data script. This results that every node getting extra label, in our case - this label is based on the current AZ. Basically, we need certain network customization in every AZ, more precisely, for every subnet in every AZ, thus we have several different eni configs. Here's the sample userdata:
--kubelet-extra-args "--node-labels=k8s.amazonaws.com/eniConfig=eni-config-$SUFFIX"

Then during provision of new node, check logs of aws-node pod. It should contain proper config name, coming from node labels.

Resource Specs and Logs

Here's the sample logs with wrong behavior:
{"level":"info","ts":"2022-03-23T00:27:37.668Z","caller":"ipamd/ipamd.go:795","msg":"Found ENI Config Name: default"}
{"level":"error","ts":"2022-03-23T00:27:37.769Z","caller":"ipamd/ipamd.go:795","msg":"error while retrieving eniconfig: ENIConfig.crd.k8s.amazonaws.com "default" not found"}
{"level":"error","ts":"2022-03-23T00:27:37.769Z","caller":"ipamd/ipamd.go:769","msg":"Failed to get pod ENI config"}

Sample expected logs:
{"level":"info","ts":"2022-04-13T17:51:01.444Z","caller":"ipamd/ipamd.go:795","msg":"Found ENI Config Name: eni-config-customized-1"}

Note that if I put that label into provisioner, then it can be discovered just fine. But, unforunately, then I'd have to create as many provisioners as I have subnets. Is that right behavior?

[tzneal]

@yerenkow Do you use this tag to force pods to schedule to particular zones?

[yerenkow]

What we want is karpenter to pick up any availability zone, best suitable at the moment, and not specify zones in the pods requirements.
However, for every zone we have certain networking setup, that comes from node' AZ.
This label is not used for scheduling our pods, but this label is being read by aws-node pod, to figure out proper network config (cni/eni).

[suket22]

Does the label show up on your node eventually?
When Karpenter creates an EC2 instance, it also creates a node object in your cluster with a set of labels it expects. This wouldn't include any args you've specified in your launch template since it only includes the labels that your pod requirements necessitate.

When the kubelet finally comes up, it should report the k8s.amazonaws.com/eniConfig=eni-config-$SUFFIX label to the API server and the labels should get merged into the node object that's already been created.

[yerenkow]

Once node is ready, aws-node already completed setup (and failed to properly init networking), pod with payload is assigned and failed b/c of no network.
I've verified that at this moment "describe node" doesn't show the expected label (it's simply not there).

[tzneal]

@yerenkow We've looked into this, but for now it seems like the only solution is to use a provisioner per custom label as you had suggested earlier. It appears that since we create the Node object initially, when kubelet starts up it will only merge in a fixed set of label names.

[github-actions]

Labeled for closure due to inactivity in 10 days.

[tonylsw]

See the same issue also when using Karpenter with ENIConfig, what we like to see is for Karpenter appends labels in provisioner to the existing labels (created from LT userdata), instead of replacing them.

[github-actions]

Labeled for closure due to inactivity in 10 days.

[jwenz723]

Maybe one option would be to allow for the availability zone to be looked up by karpenter and templated into label values which are set in a provisioner. For example, something like this:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    eniconfig: my-custom-prefix-{{ availability-zone }}

This way the label could be set using the already functional labels element in the provisioner API, but it would allow for the dynamic availability zone to be included in the label value without having 1 provisioner per AZ.

[ellistarn]

I'm missing the issue here. What's wrong with setting these special custom labels in the user data? They do eventually propagate to the node, right?

I'm wary of special casing this due to the complexity involved and the fact that we haven't seen this issue beyond this special case.

[isaac88]

Hello @ellistarn
We're currently migrating from CA to Karpenter and we are using
--kubelet-extra-args "--node-labels=storage= in our EKS Managed node groups user data launch template to add some labels depending if the underline host it's an EBS instance or and Instance Store.
We're using that label for node affinity porpuse.
Since Karpenter doesn't propagate the --kubelet-extra-args "--node-labels=storage= we have an issue trying to migrate to Karpenter.
In my humble opinion I think that feature should be really useful to have in Karpenter.

[ellistarn]

The challenge is that we can't know about the dynamic labels before the node comes online. We have well known labels to support this type of use case: https://karpenter.sh/v0.18.1/aws/instance-types/. Perhaps we need to expose storage type (ebs, instance-store, etc).

I'm not sure what the alternative is from an implementation perspective.

[jonathan-innis]

@yerenkow, we have a design doc up now that should address this issue as well of as a host of other issues: #2944

[jonathan-innis]

@yerenkow @isaac88 @jwenz723 @tonylsw We should have resolved this issue with the release of v0.28.0-rc.1. Have you been able to try out the release candidate and let us know if you are still seeing this issue.

[isaac88]

Nice @jonathan-innis, I'll test v0.28.0, and I'll let you know.

[isaac88]

Hello @jonathan-innis I confirm that the latest release v0.28.0 fix that issue.
Thanks a lot.