Kustomize command to add environment variables to containers in a kustomization

[guibirow]

Some applications allow configuration to be defined via Environment Variables. When deploying the yaml file with customize, we need to define these environment variables into the deployment.yaml file and pass the values accordingly.

In many cases, when the values are optional, we shouldn't need to provide any values or define the variables into the deployment.

Would make much more convenient if we could set these environment variable using customize commands instead of filling the deployment.yaml with all possible parameters.

Example:
Instead of defining the file like this:

kind: Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
spec:
  template:
    spec:
      containers:
      - name: wordpress
        env:           // Just one for simplicity, this could have a long list of optional variables 
        - name: WORDPRESS_DB_HOST
          value: $(MYSQL_SERVICE)

We could define like this:

kind: Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
spec:
  template:
    spec:
      containers:
      - name: wordpress

And inject environment variables with:

Specific container:

kustomize edit set env WORDPRESS_DB_HOST=$(MYSQL_SERVICE) container=wordpress

for specific deployment

kustomize edit set env WORDPRESS_DB_HOST=$(MYSQL_SERVICE) deployment=wordpress

for a specific container in a deployment:

kustomize edit set env WORDPRESS_DB_HOST=$(MYSQL_SERVICE) deployment=wordpress container=wordpress

For all containers in the kustomization

kustomize edit set env WORDPRESS_DB_HOST=$(MYSQL_SERVICE)

This will be useful to provide Environment Variables to containers dynamically at deployment time without setup the yaml file in advance.

[omninonsense]

@guibirow Hi! Would ConfigMapGenerator and SecretGenerator not suffice here? If so, why not?

[guibirow]

Probably yes,
I had a look at it but couldn't identify how would it work to create the configMap or secret keys at deployment time using a value available only during deployment(e.g: environment variables or output from other commands)

This is why I suggested this approach, given the simplicity.

Is there any command similar to this to create thes configMaps?

[omninonsense]

It's not very well documented (if at all?), but the KvLoader used by both generators defers to the execution context's environment while parsing env files when the variable name isn't being set, so if you have an env file with this contents (see here)

FOO=lalala
WORDPRESS_DB_HOST

the WORDPRESS_DB_HOST variable would be read from the environment by kustomize.

[guibirow]

Nice one, I will try that, thanks.

The only downside is the keys still need to be defined in advance and the names must match the keys throughout, but is already a good start.

Also seems a better idea than setting the environment variables in the containers.

I will reformulate the issue, would still be useful to have a command line to create these keys at deployment time, so we can adjust the naming according to our needs.

[omninonsense]

The only downside is the keys still need to be defined in advance and the names must match the keys throughout, but is already a good start.

From rereading your example it's a bit unclear what the effect of running it would be. Can you post an example kustomization and the desired outcome after running your command? I think I misunderstood it initially.

kustomize edit set env WORDPRESS_DB_HOST=$(MYSQL_SERVICE) container=wordpress

You can't avoid defining the names in advance at some point. It'll have to be a constant name somewhere down chain, if even in code. I personally don't find the ability to rename/alias or define them via kustomize that useful or practical. To me, it makes the situation more confusing, because it makes it harder to "follow" what's what or what's coming from where.

If this is really necessary for some reason—situation that comes to mind is no control over the sourcode (in this case WordPress), and no control over the original names of the variables (let's say they're already used by other stuff in the CI pipeline and they're provided by the CI)—the aliasing could still be done at the environment level, for example:

export WORDPRESS_DB_HOST=$(MYSQL_SERVICE)
kustomize build ./wordpress | kubectl apply -f -

or if you prefer to inline it:

WORDPRESS_DB_HOST=$(MYSQL_SERVICE) kustomize build ./wordpress | kubectl apply -f -

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[guibirow]

Just an update on this issue.
I ended adding tokens to the file and doing envsubst
Much simpler and avoid the need to map the env-var to the same name as the container env var names.

So I can do like below

FOO=$(MY_FOO)
WORDPRESS_DB_HOST=$(MY_WP_DB)

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[HariSekhon]

This would have been useful to be able to add say annotation to CI deployments via this field:

kubernetes.io/change-cause="deployed build $hashref"

Otherwise the kubectl rollout history has blank change-cause <none>, which would be useful if populated in case you want to track things in the kubectl CLI like when deployments happened, or roll back quickly via the CLI.

Ideally this would be even better if the change-cause could capture the GMT timestamp and a templated / environment variable message

[afirth]

dup of #388, see also kubernetes-sigs/cli-experimental#66

[boarder7395]

For others reading this:

Im not sure if this is the intention but I have found this would work:

export WORDPRESS_DB_HOST=$(MYSQL_SERVICE)
kustomize build ./wordpress | kubectl apply -f -

except the environment variable cannot have underscores otherwise the replacement does not happen. So instead use:
env.yaml

---

FOO=lalala
WORDPRESSDBHOST

---

export WORDPRESSDBHOST=$(MYSQL_SERVICE)
kustomize build ./wordpress | kubectl apply -f -

Also pay specific attention to the lack of equal after WORDPRESSDBHOST because if an equal exists it will not pull from the environment variables.

[boarder7395]

Update on the above:

I was actually using kfctl with kustomize when I ran into the underscore issue with environment variables. Using kubectl kustomize, or kustomize directly does not look to have this issue. So disregard the above unless you are using kfctl to build your kustomize manifests.

[afirth]

this is finally documented in kubernetes/website#30348

[joebowbeer]

this is finally documented in kubernetes/website#30348

but then reverted eight months later to fix kubernetes/website#35669

[confiq]

they have good point,

Other tools (sed, jinja, erb, envsubst, kafka, helm, ksonnet, etc.) provide varying degrees of unstructured editting and/or embedded languages, and can be used instead of, or in a pipe with, kustomize. If you want to go all-in on configuration as a language, consider cue.

kustomize is going to stick to YAML in / YAML out.

https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#build-time-side-effects-from-cli-args-or-env-variables

and I was looking for a way to get out of the YAML DRY principle...

At this point, I'm really looking for better alternatives like cdk8s.io so I won't be sticking with YAML

[cyprianbergoniatmo]

this is finally documented in kubernetes/website#30348

but then reverted eight months later to fix kubernetes/website#35669

If you look at the source code, https://github.com/kubernetes-sigs/kustomize/blob/master/api/kv/kv.go#L163-L169
It looks like that was a bug that will be deprecated