kustomize 5 build returns error when labels on a resource is null

[pavansokkenagaraj]

What happened?

kustomize build(V5.0.0) with patches and transformer on a resource having labels as null returns error:

Error: considering field 'metadata/labels' of object Job.v1.batch/security-job.[noNs]: wrong node kind: expected MappingNode but got ScalarNode: node contents:
null

What did you expect to happen?

expected: kustomize build to

merge the resources and resulting yaml to be returned.

kustomize v4.5.7 build returned the results when labels: null [as expected]

If I remove labels: null from resources the kustomize build works [but should work with labels:null]

How can we reproduce it (as minimally and precisely as possible)?

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- resources.yaml
transformers:
- label-transformer.yaml
patches:
- path: pullsecrets.yaml
# patchesStrategicMerge:
#   - pullsecrets.yaml

# resources.yaml
apiVersion: batch/v1
kind: Job
metadata:
  labels: null
  name: security-job
spec:
  backoffLimit: 3
  template:
    metadata:
      name: security-test
    spec:
      containers:
      - name: hook-test

# label-transformer.yaml
apiVersion: builtin
kind: LabelTransformer
metadata:
  name: label-transformer
labels:
  test.io/name: label-transformer
fieldSpecs:
- path: metadata/labels
  create: true

# pullsecrets.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: security-job
spec:
  template:
    spec:
      imagePullSecrets:
      - name: multi-namespace-yeti-registry

Expected output

➜ kustomize4 build

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    test.io/name: label-transformer
  name: security-job
spec:
  backoffLimit: 3
  template:
    metadata:
      name: security-test
    spec:
      containers:
      - name: hook-test
      imagePullSecrets:
      - name: multi-namespace-yeti-registry

Actual output

➜ kustomize build

Error: considering field 'metadata/labels' of object Job.v1.batch/security-job.[noNs]: wrong node kind: expected MappingNode but got ScalarNode: node contents:
null

Kustomize version

V5.0.0

Operating system

Linux

[KnVerey]

What is the intention of including labels: null in the base resource? This change is very likely caused by a fix we made to the handling of null values in base resources: #4890 . Previously, they were incorrectly treated as a deletion directive. Now, they are retained, and in this case are not the correct type for the field (empty value should be {} for a map).

/triage needs-information

[sgalsaleh]

@KnVerey here's another example where Kustomize 5 isn't able to process null values that Kustomize 4 and Kubernetes (e.g. kubectl apply) are okay with, which is initContainers: null:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      initContainers: null
      containers:
      - name: nginx
        image: nginx

pull-secret.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    spec:
      imagePullSecrets:
      - name: example-pull-secret

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesStrategicMerge:
- pull-secret.yaml
resources:
- deployment.yaml

Running kustomize build:

$ kustomize build kustomize-example/
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Error: updating name reference in 'spec/template/spec/initContainers/env/valueFrom/configMapKeyRef/name' field of 'Deployment.v1.apps/nginx.[noNs]': considering field 'spec/template/spec/initContainers/env/valueFrom/configMapKeyRef/name' of object Deployment.v1.apps/nginx.[noNs]: expected sequence or mapping node

[sgalsaleh]

For example, there are helm charts out there that produce null values depending on the result of templating.

[sgalsaleh]

@KnVerey any plans to address this issue? this unfortunately breaks some of the Helm charts our customers are using in production and it's not easily workaround-able without breaking existing functionality.

[obfuscurity]

Any updates?

[obfuscurity]

Is anyone able to look at this issue? This remains a blocker to our upgrading to 5.x.

[cbodonnell]

Hi @KnVerey, I have some more interesting findings as it relates to this issue. Apologies in advance for the lengthy comment...

It appears that kustomize will actually add null to a field if it is empty. For example, some resources in community Helm charts may look like the following after rendering templates (notice the empty labels):

resource.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp
  labels:
data:
  key: value

If I simply run this resource through kustomize without any transformations:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- resource.yaml

The build output is the following:

$ kustomize build
apiVersion: v1
data:
  key: value
kind: ConfigMap
metadata:
  labels: null
  name: myapp

If I then try to apply a label transformer and patch to this output, it will fail with the aforementioned error. However, it will not if I use the original resource with the empty labels field. Additionally, it seems to work fine if I leave out either the label transformer or the patch, which also seems odd.

new-resource.yaml:

apiVersion: v1
data:
  key: value
kind: ConfigMap
metadata:
  labels: null
  name: myapp

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- new-resource.yaml
transformers:
- label-transformer.yaml
patches:
- path: patch.yaml

label-transformer.yaml:

apiVersion: builtin
kind: LabelTransformer
metadata:
  name: label-transformer
labels:
  test.io/name: label-transformer
fieldSpecs:
- path: metadata/labels
  create: true

patch.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp
data:
  newkey: newvalue

Result:

$ kustomize build
Error: considering field 'metadata/labels' of object ConfigMap.v1.[noGrp]/myapp.[noNs]: wrong node kind: expected MappingNode but got ScalarNode: node contents:
null

After swapping out new-resource.yaml for the original resource.yaml with the empty field:

$ kustomize build
apiVersion: v1
data:
  key: value
  newkey: newvalue
kind: ConfigMap
metadata:
  labels:
    test.io/name: label-transformer
  name: myapp

[obfuscurity]

Any updates from maintainers?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.