Support for managing env vars via kustomize

[djones6]

Feature Request

Is your feature request related to a problem?

If an application accepts configuration via a number of environment variables, these variables may have hard-coded defaults in the spec/env section of app-deploy.yaml that are suitable for (say) a development environment.

Kustomize can be used to patch values in the deployment to tailor it for different environments (eg. test or production). Kustomize has built-in knowledge of the structure of standard Kubernetes types (Deployment, Service etc) and the env section of a vanilla Deployment is recognized as a map which is keyed by the name field. It can therefore be patched either to add new entries or replace the values for existing ones.

However, Kustomize has no knowledge for custom resource types. As a result, trying to apply a Strategic Merge Patch to the env list of an AppsodyApplication CR will result in the entire list being replaced. Similarly, an SMP cannot be used to update an existing value in place.

A workaround is to use JSON patches (using kustomize's patchesJson6902 directive rather than patchesStrategicMerge), although these are also unable to refer to named elements of the list, so are limited to append/prepend, or replacing an entry by numeric index.

Another option is for all the 'dynamic' values in the env list (ie. values that may change at deploy time) to be references to ConfigMaps or Secrets, and for kustomize to define the contents of those resources instead. This would avoid needing to patch the env list at all and is the route I'm going to explore next. But I imagine it could cause surprise if patching does not behave the same as with a vanilla Deployment.

Describe the solution you'd like

To be able to apply patches to map types in the AppsodyApplication CR the same way that I can for a vanilla Kubernetes Deployment resource.

Having read around a bit, I found discussion on this topic in kubernetes-sigs/kustomize#1510 - from which I understand that it is not yet possible to educate kustomize on the structure of a custom resource, although there is an issue open for that support here: kubernetes-sigs/kustomize#2339

[arthurdm]

thanks for the feature request @djones6

Do you have any outlook on when that kustomize support will be coming in?

My fear is that we have a few other places that a person could also want to "kustomize", such as volumes, etc. So if we had a special extra section for env vars we would likely need to address the other spots too - and all that would be not needed anymore once the native CR kustomize support is available =)

For GitOps, is it reasonable to assume that for now each kustomize patch would contain the complete set of env vars, even if that means duplicating some key/values across environments?

Another strategy - create a ConfigMap for your "static" env vars, and use envFrom to point to that. This leaves you free to override the env array via kustomize patches.