Figure out how to represent per-environment customizations, for dev and prod

[mortent]

Keeping in mind blast radius and permissions

[delve]

So I have a functioning POC separating env config from cross-env config that looks roughly like

kind: Kptfile
...
pipeline:
  mutators:
    - image: set-image
      configPath: ./env-configs/image.yaml
    - image: set-labels
      configPath: ./env-configs/labels.yaml
    - image: set-labels
      configPath: ./global-configs/labels.yaml
... etc

with relevant function configurations as described in the pipeline. This lives in a repo with environments isolated by directory. The purpose being to enable automated promotion without clobbering environment specific details.

I also wonder if it's possible to put the pipeline itself into its own resource to enable something like:

pipelines:
  - sourceFile: ./env-mutators.yaml
  - sourceFile: ./global-mutators.yaml
  - sourceFile: ./env-validators.yaml
  - sourceFile: ./global-validators.yaml

This would allow, for example, application of stricter policy validation in a prod environment. I know we could edit the package in place in prod to do that, but that makes promotion tricky from lower environments.

[bgrant0607]

Thanks for the example, @delve!

Each variant requires variant-specific inputs. If the image tag or digest is one of those inputs, how did you want to propagate a new value across environments?

[delve]

I'm expecting CI automation to copy promotable changes across environment directories. In the overly-simplified example above I should have had image.yaml in a different subdirectory. Something like ./promotable-configs maybe.

Once promotion is approved (either via automated testing or a PR process) the CI copies ./dev/promotable-configs to ./prod/promotable-configs.yaml. And then gitops takes over pushes the change to the cluster.

[bgrant0607]

@delve What do you think about the approach of creating a package with all common attributes and the function pipeline (a "blueprint"), clone one copy of it per environment via kpt pkg get, and change environment-specific attributes in each copy (aka variant)? In order to promote an image tag/digest, it would be changed in the blueprint, and then kpt pkg update would be invoked on each variant.

We're trying to make those types of updates easier with the package orchestrator. An example of a bulk upgrade flow can be seen in the prototype demo: https://www.youtube.com/watch?v=d_iV22_6nAM

[delve]

@bgrant0607 I'm not opposed to that approach, but the org I'm working for is approaching blueprints as generic building blocks agnostic of the deployed code. Instead of 10 blueprints, one per app, we're looking at 3 blueprints, each a composable part, where the variant supplies app details like which image to use.

Under that arrangement it's necessary to maintain configuration data separate from the package. I'm working on implementing a wrapper pattern for this.

[bgrant0607]

@delve I expect kpt to work with multiple levels of derivation, such as generic blueprint -> multi-environment app blueprint -> specific environment deployment package

[bgrant0607]

We'll want a way to be able to set properties in common per environment across multiple applications -- mixins effectively.

[leeren]

Has any progress been made on making propagating variant-specific inputs to generic blueprints with kpt easier? @delve curious what you ended up doing.

I agree that isolating environment-specific config settings by folder and having CI selectively run them seems like the way to go. Is it possible for the kptfile pipeline itself to be templatable to make this easier?