🌱 Generate cluster templates with kustomize

[mdbooth]

What this PR does / why we need it:

This PR changes the way we generate our cluster templates for both release and e2e tests to use kustomize. We now have the following dependency structure between templates:

graph TB
  subgraph Release
    release/without-lb([Without LoadBalancer]) --> release/default([Default])
    release/external-cloud-provider([External Cloud Provider]) --> release/default
  end

  subgraph E2E Tests
    test/default([Default]) --> release/default
    test/external-cloud-provider([External Cloud Provider]) --> release/external-cloud-provider
    test/without-lb([Without LoadBalancer]) --> release/without-lb
    test/multi-az([Multi AZ]) --> test/default
    test/multi-network([Multi Network]) --> test/default
  end

Loading

Additionally, all the E2E templates now use a common set of patches to uniformly apply configuration we use in tests but not in release. This means that any change in the release templates is now reflected automatically in the E2E tests, which gives us a level of test coverage for changes to the release templates.

That said, in order to keep this PR simple the templates produced by this PR are identical to the previous templates except in the order of the objects and fields. The order difference is an artifact of using kustomize: it picks a consistent order according to its own rules. I have verified that the templates are identical by the following:

For every existing template in /templates and /test/e2e/data/infrastructure-openstack I generated a kustomization.yaml containing only a resources section specifying only the original template file, and then I built the kustomization. i.e. This is the original template whose only transformation is the re-ordering done by kustomize. I then generated templates from the new kustomize and verified that there were no diffs.

Release templates are generated by a new make target: templates.
E2E templates are generated by a new make target: e2e-templates.

I have created directories for v1alpha5 and v1alpha6, which are currently identical except for apiVersion, and we should consider adding a directory for v1alpha4 in the future as long as we support it. The intention here is to for reuse when adding tests for supported previous versions.

I have used strategic merge patches in the release templates because I find them more readable. However, in the e2e templates I have used a json patch everywhere we reference an OpenStack object. This allows the patches to be reusable across apiVersions.

/hold

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name	Link
🔨 Latest commit	122d7c1
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-openstack/deploys/62bc3094773bf500090d67d8
😎 Deploy Preview	https://deploy-preview-1271--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[apricote]

What is the purpose of this target?

[mdbooth]

I could not find any way to make the target depend on all files in a directory. A target that depends on a .PHONY target will always be rebuilt. The targets which use FORCE are pattern targets, and AFAIK there is no way to make them directly .PHONY.

[apricote]

This fails for me with:

$ make templates
hack/tools/bin/kustomize build "kustomize/v1alpha6/default" > "templates/cluster-template.yaml"
bash: line 1: hack/tools/bin/kustomize: No such file or directory
make: *** [Makefile:404: templates/cluster-template.yaml] Error 127
make: *** Deleting file 'templates/cluster-template.yaml'

I would suggest we add the $(KUSTOMIZE) target as a dependency, so the binary is always available

Suggested change

	templates/cluster-template.yaml: kustomize/v1alpha6/default FORCE
	templates/cluster-template.yaml: kustomize/v1alpha6/default FORCE $(KUSTOMIZE)

The same applies to targets templates/cluster-template-%.yaml, $(E2E_TEMPLATES_DIR)/cluster-template.yaml and $(E2E_TEMPLATES_DIR)/cluster-template-%.yaml

[mdbooth]

I've updated with this change.

[apricote]

/lgtm

[tobiasgiese]

Also validated the changes via

git clone https://github.com/kubernetes-sigs/cluster-api-provider-openstack/ /tmp/cluster-api-provider-openstack
for template in templates/*yaml; do diff -u <(sort $template) <(sort /tmp/cluster-api-provider-openstack/$template); done

Noticed only some quoting and list syntax changes

Edit: btw, there is also a yaml differ called dyff. But as there are multiple yamls inside the manifest dyff is not able to do its proper job 😿

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mdbooth, tobiasgiese

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mdbooth,tobiasgiese]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tobiasgiese]

Edit: btw, there is also a yaml differ called dyff. But as there are multiple yamls inside the manifest dyff is not able to do its proper job

But there is ksort.
Combined I couldn't find any changes

$ for template in templates/*yaml; do dyff between --omit-header <(ksort -f $template) <(ksort -f /tmp/cluster-api-provider-openstack/$template); done
$

[mdbooth]

Edit: btw, there is also a yaml differ called dyff. But as there are multiple yamls inside the manifest dyff is not able to do its proper job

But there is ksort. Combined I couldn't find any changes

$ for template in templates/*yaml; do dyff between --omit-header <(ksort -f $template) <(ksort -f /tmp/cluster-api-provider-openstack/$template); done
$

I did something like this against an unmodified repo to create a new directory (k) containing original templates which have been sorted by kustomize:

mkdir k
for i in cluster-template*.yaml; do
  echo -e "resources:\n- $i" > kustomization.yaml
  kustomize build . > k/$i
end

Then I diffed the new generated templates against the kustomize-sorted original templates. They are literally identical.

Because the original doesn't apply any patches I think this even avoids kustomize pitfalls involving incorrect API definitions, so I'm fairly confident this is a robust test.

[apricote]

/lgtm

[mdbooth]

/hold cancel