Using replacements with a commonLabel as a source

[dharmeshgordhan]

Hi,

Any suggestions on how I can use replacements to read the commonLabels value? I'm getting an error at the moment: Error: nothing selected by Kustomization.[noVer].[noGrp]/[noName].[noNs]:commonLabels.[pipeline/branch].

I'm looking to have my app pod, and database pod deployed on the same node. I'm using podAffinity to do this. My deployment method means application feature branches get deployed, and each deployment has the unique label pipeline/branch. I'm looking to use pipeline/branch label as the labelSelector for podAffinity. I wanted to use replacements to take the label from my kustomization.yaml and replaceme the BRANCH_NAME is my deployment.

sample of my kustomize manifest:

# app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: tier
                operator: In
                values:
                  - database
        - labelSelector:
            matchExpressions:
              - key: pipeline/branch
                operator: In
                values:
                  - BRANCH_NAME

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: staging
commonLabels:
  pipeline/branch: feature/myBranch
replacements:
  - source:
      kind: Kustomization
      name: staging
      fieldPath: commonLabels.[pipeline/branch]
    targets:
      - select:
          kind: Deployment
          name: app
        fieldPaths:
          - spec.affinity.podAffinity.requiredDuringSchedulingIgnoredDuringExecution.[1].matchExpressions.[key=pipeline/branch].values.0

[k8s-ci-robot]

@dharmeshgordhan: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[KnVerey]

It isn't possible to target the Kustomization itself using the replacements transformer--it only operates on the resource data that the Kustomization has already produced. Would you be able to instead target the label field on the app Deployment in question? Since the label transformer runs first, I would expect the value you want to be there when the replacement transformer runs.

/triage needs-information
/kind feature

[dharmeshgordhan]

I had a quick go, but it threw an error. I will try more diligently in a couple of days - thought I'd share what I had so far.

`metadata.labels.pipeline/branch` is missing for replacement source Deployment.[noVer].[noGrp]/db.[noNs]

I had a look at the output without the replacements, and the attribute looks like it's there. Trying with square brackets around the label for the replacement source threw:

`error looking up replacement source: list path element must contain fieldName=fieldValue for element to match`

Here's a sample of the deployment object output from kustomize:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    pipeline/branch: bloop
  name: db

Here's the replacements:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: staging
commonLabels:
  pipeline/branch: bloop
replacements:
  - source:
      kind: Deployment
      name: db
      fieldPath: metadata.labels.pipeline/branch
    targets:
      - select:
          kind: Deployment
          name: app
        fieldPaths:
          - spec.affinity.podAffinity.requiredDuringSchedulingIgnoredDuringExecution.[1].matchExpressions.[key=pipeline/branch].values.0

[dharmeshgordhan]

Okay! Got it working :) Not toooo sure why it didn't work the first time.

Thanks!

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  pipeline/branch: bloop
replacements:
  - source:
      kind: Deployment
      name: db
      fieldPath: metadata.labels.pipeline/branch
    targets:
      - select:
          kind: Deployment
          name: app
        fieldPaths:          
          - spec.template.spec.affinity.podAffinity.requiredDuringSchedulingIgnoredDuringExecution.1.labelSelector.matchExpressions.0.values.0

[dharmeshgordhan]

Oh I just realised that it doesn't work when I have the common label in my overlay, and the replacements in the base.

But if I have the common label in my overlay and the replacements in the same overlay it works.

Is that expected? If so, the downside is I'd need to duplicate my replacements blocks into the overlay kustomization.yaml.

[k8s-ci-robot]

@dharmeshgordhan: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[KnVerey]

Oh I just realised that it doesn't work when I have the common label in my overlay, and the replacements in the base.

Yes, that is expected. When you reference a base in the resources field, it is entirely evaluated and its resources are added to any others you've specified in the overlay, and any transformations you specify in the overlay will run after that.

[webwurst]

Hi! I just tried to use values from commonLabels with some replacements which were pulled in as components in an overlay. It seems the replacements take place in the overlay before the commonLabels of the overlay are applied. So in the end I get manifests with the labels defined in the overlay, but the values that got replaced contain the values from the commonLabels defined in the base. This feels a bit odd since the replacement is meant to align the label value with some target fields in the manifest and now they are different :) I understand this is just a matter of the ordering of all the transformers. Is this something that can be fixed by changing the order? Or will this lead to other problems? I'll try to find an different way to address my issue. But would be interested in your opinion.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
  cluster.x-k8s.io/cluster-name: foofoo
  testing.io/cluster: f0001
  testing.io/organization: lab-for-foo
  release.testing.io/version: 3.0.2

resources:
- ../../../base/cluster
- ./control-plane

components:
- ../../../components/replace-organization-name
- ../../../components/replace-cluster-name
- ../../../components/replace-control-plane-name

[KnVerey]

I agree with you that it is an ordering issue. Since Kustomize is quite generic and can be used to build configuration sets in many different ways, I expect there is no perfect ordering we can choose. As a rule we try to put generation-oriented functionality earlier and transformation-oriented functionality later, but components sit squarely in the middle--they're essentially a resources field that is allowed to operate on the in-flight state instead of only add to it. In the long term, I hope to have a new Composition type that will allow completely arbitrary ordering of our features: KEP 2299. One of the biggest pieces required for that is #4323, which would help with reordering within Kustomization as well. It's still part of the long-term plan, but it is tricky work and we haven't had the resources to prioritize it.

[webwurst]

Thank you for the prompt answer :) Looking forward to the Composition API in the long run then.

[KnVerey]

/close
/triage resolved

I believe my previous comments addressed this.

[k8s-ci-robot]

@KnVerey: Closing this issue.

In response to this:

/close
/triage resolved

I believe my previous comments addressed this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.