Figure out how to represent per-environment customizations, for dev and prod

[liamfallon]

Original issue URL: kptdev/kpt#3280
Original issue user: https://github.com/mortent
Original issue created at: 2022-05-27T23:09:17Z
Original issue last updated at: 2023-01-17T06:29:49Z
Original issue body: Keeping in mind blast radius and permissions

Original issue comments:
Comment user: https://github.com/delve
Comment created at: 2022-06-30T15:10:21Z
Comment last updated at: 2022-06-30T15:10:21Z
Comment body: So I have a functioning POC separating env config from cross-env config that looks roughly like

kind: Kptfile
...
pipeline:
  mutators:
    - image: set-image
      configPath: ./env-configs/image.yaml
    - image: set-labels
      configPath: ./env-configs/labels.yaml
    - image: set-labels
      configPath: ./global-configs/labels.yaml
... etc

with relevant function configurations as described in the pipeline. This lives in a repo with environments isolated by directory. The purpose being to enable automated promotion without clobbering environment specific details.

I also wonder if it's possible to put the pipeline itself into its own resource to enable something like:

pipelines:
  - sourceFile: ./env-mutators.yaml
  - sourceFile: ./global-mutators.yaml
  - sourceFile: ./env-validators.yaml
  - sourceFile: ./global-validators.yaml

This would allow, for example, application of stricter policy validation in a prod environment. I know we could edit the package in place in prod to do that, but that makes promotion tricky from lower environments.

Comment user: https://github.com/bgrant0607
Comment created at: 2022-06-30T15:25:12Z
Comment last updated at: 2022-06-30T15:25:12Z
Comment body: Thanks for the example, @delve!

Each variant requires variant-specific inputs. If the image tag or digest is one of those inputs, how did you want to propagate a new value across environments?

Comment user: https://github.com/delve
Comment created at: 2022-06-30T15:52:59Z
Comment last updated at: 2022-06-30T15:53:53Z
Comment body: I'm expecting CI automation to copy promotable changes across environment directories. In the overly-simplified example above I should have had image.yaml in a different subdirectory. Something like ./promotable-configs maybe.

Once promotion is approved (either via automated testing or a PR process) the CI copies ./dev/promotable-configs to ./prod/promotable-configs.yaml. And then gitops takes over pushes the change to the cluster.

Comment user: https://github.com/bgrant0607
Comment created at: 2022-06-30T16:33:24Z
Comment last updated at: 2022-06-30T16:33:24Z
Comment body: @delve What do you think about the approach of creating a package with all common attributes and the function pipeline (a "blueprint"), clone one copy of it per environment via kpt pkg get, and change environment-specific attributes in each copy (aka variant)? In order to promote an image tag/digest, it would be changed in the blueprint, and then kpt pkg update would be invoked on each variant.

We're trying to make those types of updates easier with the package orchestrator. An example of a bulk upgrade flow can be seen in the prototype demo: https://www.youtube.com/watch?v=d_iV22_6nAM

Comment user: https://github.com/delve
Comment created at: 2022-07-08T12:33:29Z
Comment last updated at: 2022-07-08T12:33:29Z
Comment body: @bgrant0607 I'm not opposed to that approach, but the org I'm working for is approaching blueprints as generic building blocks agnostic of the deployed code. Instead of 10 blueprints, one per app, we're looking at 3 blueprints, each a composable part, where the variant supplies app details like which image to use.

Under that arrangement it's necessary to maintain configuration data separate from the package. I'm working on implementing a wrapper pattern for this.

Comment user: https://github.com/bgrant0607
Comment created at: 2022-07-08T16:58:52Z
Comment last updated at: 2022-07-08T16:58:52Z
Comment body: @delve I expect kpt to work with multiple levels of derivation, such as generic blueprint -> multi-environment app blueprint -> specific environment deployment package

Comment user: https://github.com/bgrant0607
Comment created at: 2022-07-30T00:35:48Z
Comment last updated at: 2022-07-30T00:35:48Z
Comment body: We'll want a way to be able to set properties in common per environment across multiple applications -- mixins effectively.

Comment user: https://github.com/leeren
Comment created at: 2023-01-17T06:29:49Z
Comment last updated at: 2023-01-17T06:29:49Z
Comment body: Has any progress been made on making propagating variant-specific inputs to generic blueprints with kpt easier? @delve curious what you ended up doing.

I agree that isolating environment-specific config settings by folder and having CI selectively run them seems like the way to go. Is it possible for the kptfile pipeline itself to be templatable to make this easier?