[do-not-merge] feat: kustomize proof-of-concept

[artsyjian]

PLATFORM-4488

This is a spike on Kustomize.

The motivation is to manage Hokusai Kubernetes specs (staging.yml, production.yml) better. Those specs in each Hokusai-managed project are copied from artsy-hokusai-templates repo during project initialization (via hokusai setup). Once copied, they begin to have lives of their own and evolve separately from the templates. We modify the templates from time to time, but there is no automated way to sync changes to the projects. Manual syncs are tedious, because 1) a pull request is required for every project, and 2) one cannot simply copy from the templates and paste, but must be careful not to over-write anything that is customized in the projects. Due to these difficulties, almost all of our projects (including this one) are lagging behind the templates.

And, there is too much duplication of specs. Everything is written four times: in template staging.yml, template production.yml, project staging.yml, project production.yml.

We would like to explore whether Kustomize can solve these problems. We start with this simple sandbox app and replace its staging.yml/production.yml with two directories: staging, production. Each directory contains Kustomize config to generate the Kubernetes spec appropriate for the corresponding environment. Specs for the two environments are generated from the same base spec which is fetched at build time from artsy-hokusai-templates. None of the specs are checked into this project's repo.

The base spec covers: Deployment, HorizontalPodAutoscaler, Service, Ingress. The CronJob is kept out of the base because it is very unique to each project and environment. A unkustomized.yml is used to store it and other resources that should not be modified by Kustomize.

The patching and transforms are done as specified in kustomization.yml, patch.yml, transform.yml. The modifications are mostly to insert the project's name into various fields in the Kubernetes spec, a task that Hokusai already does on staging.yml and production.yml using Jinja, so we might not even need Kustomize for it.

Production has a patch.yml that patches the HorizontalPodAutoscaler spec found in the base. The change is on minReplicas/maxReplicas so only those fields are specified. The spec in patch.yml is not a complete HorizontalPodAutoscaler spec. It does not have to be. Staging has no patch.yml because the base spec is already good enough.

To build staging and production specs, first install kustomize, then:

jian@artsy:~/code/hokusai-sandbox$ kustomize build hokusai/staging

jian@artsy:~/code/hokusai-sandbox$ kustomize build hokusai/production

Obviously, Hokusai would have to be modified to wire kustomize into it.

[kajatiger]

If this really works it would be super magical. I am not sure I understand why we need kustomize for this if Jinja and hokusai already do most of the adaption between spec and projects...?

[artsyjian]

If this really works it would be super magical. I am not sure I understand why we need kustomize for this if Jinja and hokusai already do most of the adaption between spec and projects...?

Currently we use Jinja only for variable interpolation within each project, replacing (what I am aware of): project_name, project_repo, ingress IP allowlist, and ingress config snippet. This does not help with the problem of a project's staging/production.yml drifting from artsy-hokusai-templates.

[joeyAghion]

Wow, so a "standard"/small app would probably get by with just a name prefix and image specified. Everything else would be applied "live" from the templates.

Should we use the artsy-hokusai-templates project for both project generation templates and these base specs? On the one hand, the generated YAML files will be closely related to the base files. On the other hand, the base specs will be applied to production on an ongoing, unsupervised basis, so require extra care.

It bugs me that both hokusai and kustomize can be influencing the interpolated specs. Is it possible to eliminate the hokusai templating in favor of kustomize's (more generally accepted) approach?

[artsyjian]

Wow, so a "standard"/small app would probably get by with just a name prefix and image specified. Everything else would be applied "live" from the templates.

"live", yes. does feel scary, but not sure if it's really dangerous, for example, projects been using "live" hokusai/orb for a while and no problems. i guess it depends on how solid Kustomize is.

for controlled template roll outs, we can version them and add a version number to the base URL:

bases:
- https://github.com/artsy/artsy-hokusai-templates//kustomize/base?ref=v0.0.1

when template v0.0.2 is out, we update the URL. will have to be done per project.

[artsyjian]

Should we use the artsy-hokusai-templates project for both project generation templates and these base specs?

We can have Kustomize base dir (and its ymls) in a different Github repo, but then we will have duplication between artsy-hokusai-templates and that repo. So ideally, the base ymls replace staging.yml.j2 and production.yml.j2.

[artsyjian]

It bugs me that both hokusai and kustomize can be influencing the interpolated specs. Is it possible to eliminate the hokusai templating in favor of kustomize's (more generally accepted) approach?

Hokusai (Jinja) interpolates Cloudflare IPs with values from S3 (which it can similarly do for any sensitive values). For Gravity, Hokusai also interpolates Ingress config with value found in local file. These are in additon to the standard project_name/repo interpolations Hokusai does for every project. I don't know Kustomize can take over all that, will have to check.

hokusai setup for a Kustomize'd project will have to pull down Kustomize ymls like this one. If Kustomize can somehow interpolate project-specific fields (e.g. namePrefix) then great, otherwise, the field can be namePrefix: {{ project_name-}}, and Hokusai can interpolate it during hokusai setup (done once) or every time before kustomize build.

[artsyjian]

Hokusai (Jinja) interpolates...I don't know Kustomize can take over all that, will have to check.

Kustomize can pick up key/value pairs from a local file, env, etc, but only for the limited purpose of generating k8s configmaps or secrets. Variable interpolation in Kustomize ymls (e.g. the base ymls) is not supported. There are workarounds: pre-processing ymls with Sed, a Kustomize transformer based on Sed, but none of them seem better than our current Hokusai/Jinja approach.

Kustomize is a Yaml patching tool. The Kustomize folks believe var-interpolation (templating) is not really its job.

[artsyjian]

spiked on a hokusai staging kustomize command. It first variable-interpolates all yaml files in staging dir, then calls kustomize build.

[artsyjian]

This PoC as been heavily revised as follows:

• The base spec in artsy-hokusai-templates is now multiple kustomize directories. Each directory is dedicated to a single Kubernetes resource. This makes things flexible. Projects can now cherry pick the k8s resources they actually use.
• In order to test end to end, Hokusai branch is created with the following features:
  • a fetch_base function that git clones artsy-hokusai-templates repo, and copies the base dirs to a working directory created for staging or production.
  • a render function that does a round of var interpolation on all yml files which include those under staging and production directories, and those that were cloned from artsy-hokusai-templates.
  • a kustomize command that runs kustomize build on the resulting files.
• Because of the Hokusai changes, things are simplified as follows:
  • Bases are referenced by local ../base/... directories that are created by Hokusai at Kustomize build time.
  • Transforms (namePrefix, patchesJson6902, LabelTransformer) that are for project-wide values (project name, ecr repo) are removed. They are replaced by template vars in base ymls and Kustomize ymls, and these vars are interpolated by Hokusai.

[artsyjian]

closing per ticket update.