Merge pull request #1314 from markusheinemann/master

Add recursive user manifests

README.md

@@ -178,7 +178,7 @@ As Cilium has a lot of interesting and powerful config possibilities, we give yo
 
 Cilium supports full kube-proxy replacement. Cilium runs by default in hybrid kube-proxy replacement mode. To achieve a completely kube-proxy-free cluster, set `disable_kube_proxy = true`.
 
-It is also possible to enable Hubble using `cilium_hubble_enabled = true`. In order to access the Hubble UI, you need to port-forward the Hubble UI service to your local machine. By default, you can do this by running `kubectl port-forward -n kube-system service/hubble-ui 12000:80` and then opening `http://localhost:12000` in your browser. 
+It is also possible to enable Hubble using `cilium_hubble_enabled = true`. In order to access the Hubble UI, you need to port-forward the Hubble UI service to your local machine. By default, you can do this by running `kubectl port-forward -n kube-system service/hubble-ui 12000:80` and then opening `http://localhost:12000` in your browser.
 However, it is recommended to use the [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) and [Hubble Client](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) and running the `cilium hubble ui` command.
 
 ## Scaling Nodes
@@ -293,7 +293,7 @@ If you need to install additional Helm charts or Kubernetes manifests that are n
 
 These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `extra_kustomize_parameters` variable (via a map) to the module).
 
-All files in the `extra-manifests` directory including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
+All files in the `extra-manifests` directory and its subdirectories including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
 
 See a working example in [examples/kustomization_user_deploy](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/master/examples/kustomization_user_deploy).
 

examples/kustomization_user_deploy/README.md

@@ -1,36 +1,72 @@
-# How install deploy a additional / extra stuff while terraforming the cluster
+# How to Install and Deploy Additional Resources with Terraform and Kube-Hetzner
 
-## With a `HelmChart` and `HelmChartConfig`
+Kube-Hetzner allows you to provide user-defined resources after the initial setup of the Kubernetes cluster. You can deploy additional resources using Kustomize scripts in the `extra-manifests` directory with the extension `.yaml.tpl`. These scripts are recursively copied onto the control plane and deployed with `kubectl apply -k`. The main entry point for these additional resources is the `kustomization.yaml.tpl` file. In this file, you need to list the names of other manifests without the `.tpl` extension in the resources section.
 
-This is how it worked for me, note I'm a total beginner with kustomize.<br>
-Pretty sure I butchered a lot ;)
+When you execute terraform apply, the manifests in the extra-manifests directory, including the rendered versions of the `*.yaml.tpl` files, will be automatically deployed to the cluster.
 
-### Assuming you followed the `DO not Skip` part of the installation
+## Examples
 
-In the project folder, where the `kube.tf` is located:
-1. Create a folder named `extra-manifests`.
-2. In it create a file named `kustomization.yaml.tpl` and **your** manifest file(s). Be sure to use the `resources` field, in the `kustomization.yaml` file, to define the list of resources to include in a configuration.
+Here are some examples of common use cases for deploying additional resources:
 
-## Apply the kustomized configuration
+> **Note**: When trying out the demos, make sure that the files from the demo folders are located in the `extra-manifests` directory.
 
-Assuming no errors have been made, apply this by run `terraform apply`<br>
+### Deploying Simple Resources
 
-## ReRun the kustomization (debugging)
+The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf` file.
 
-In the highly unlikely case that an actual error has occurred...<br>
-Anyway, you can rerun just the kustomization part like this:
+#### `kube.tf`
 
-```sh
-terraform apply -replace='module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]' --auto-approve
 ```
-
-Check what kustomization exists:
-
-```sh
-(⎈|dev3:default)➜ dev3-cluster (main) ✗ terraform state list | grep kustom
 ...
-module.kube-hetzner.null_resource.kustomization
-module.kube-hetzner.null_resource.kustomization_user["some-random-name.yaml.tpl"]
-module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]
+extra_kustomize_parameters = {
+  my_config_key = "somestring"
+}
 ...
 ```
+
+The variable defined in `kube.tf` can be used in any `.yaml.tpl` manifest.
+
+#### `configmap.tf`
+
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-config
+  data:
+    someConfigKey: ${sealed_secrets_crt}
+```
+
+For a full demo see the [simple-resources](simple-resources/) example.
+
+### Deploying a Helm Chart
+
+If you want to deploy a Helm chart to your cluster, you can use the [Helm Chart controller](https://docs.k3s.io/helm) included in K3s. The Helm Chart controller provides the CRDs `HelmChart` and `HelmChartConfig`.
+
+For a full demo see the [helm-chart](helm-chart/) example.
+
+### Multiple Namespaces
+
+In more complex use cases, you may want to deploy to multiple namespaces with a common base. Kustomize supports this behavior, and it can be since Kube-Hetzner is considering all subdirectories of `extra-manifests`.
+
+For a full demo see the [multiple-namespaces](multiple-namespaces/) example.
+
+## Debugging
+
+To check the existing kustomization, you can run the following command:
+
+```
+$ terraform state list | grep kustom
+  ...
+  module.kube-hetzner.null_resource.kustomization
+  module.kube-hetzner.null_resource.kustomization_user["demo-config-map.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["demo-pod.yaml.tpl"]
+  module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]
+  ...
+```
+
+If you want to rerun just the kustomization part, you can use the following command:
+
+```
+terraform apply -replace='module.kube-hetzner.null_resource.kustomization_user["kustomization.yaml.tpl"]' --auto-approve
+```

examples/kustomization_user_deploy/helm-chart/helm-chart.yaml.tpl

@@ -0,0 +1,12 @@
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: argocd
+  namespace: argocd
+spec:
+  repo: https://argoproj.github.io/argo-helm
+  chart: argo-cd
+  targetNamespace: argocd
+  valuesContent: |-
+    global:
+      domain: argocd.example.com

examples/kustomization_user_deploy/helm-chart/kustomize.yaml.tpl

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - helm-chart.yaml

examples/kustomization_user_deploy/helm-chart/namespace.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: argocd

examples/kustomization_user_deploy/mutliple-namespaces/base/pod.yaml.tpl

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: myapp-pod
+  labels:
+    app: myapp
+spec:
+  containers:
+    - name: nginx
+      image: nginx:1.7.9

examples/kustomization_user_deploy/extra-manifests/kustomization.yaml.tpl → examples/kustomization_user_deploy/mutliple-namespaces/kustomization.yaml.tpl

@@ -2,5 +2,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
 resources:
-  - some-random-name.yaml
-  - cert-manager-webhook-inwx.yaml
+  - namespace-a
+  - namespace-b

examples/kustomization_user_deploy/mutliple-namespaces/namespace-a/kustomization.yaml.tpl

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - ../base
+namespace: namespace-a

examples/kustomization_user_deploy/mutliple-namespaces/namespace-a/namespace-a.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: namespace-a

examples/kustomization_user_deploy/mutliple-namespaces/namespace-b/kustomization.yaml.tpl

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - ../base
+namespace: namespace-b

examples/kustomization_user_deploy/mutliple-namespaces/namespace-b/namespace-b.yaml.tpl

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: namespace-b

examples/kustomization_user_deploy/simple-resources/demo-config-map.yaml.tpl

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-config
+data:
+  someConfigKey: ${sealed_secrets_crt}

examples/kustomization_user_deploy/simple-resources/demo-pod.yml.tpl

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: demo
+spec:
+  containers:
+    - name: demo-container
+      image: registry.k8s.io/busybox
+      command: [ "/bin/sh", "-c", "env" ]
+      env:
+        - name: DEMO_ENVIRONEMNT_VARIABLE
+          valueFrom:
+            configMapKeyRef:
+              name: demo-config
+              key: someConfigKey
+  restartPolicy: Never

examples/kustomization_user_deploy/simple-resources/kustomization.yaml.tpl

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - demo-config-map.yaml.tpl

kube.tf.example

@@ -806,7 +806,7 @@ module "kube-hetzner" {
   # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
   # extra_kustomize_parameters={}
 
-  # See an working example for just a manifest.yaml, a HelmChart and a HelmChartConfig examples/kustomization_user_deploy/README.md
+  # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
 
   # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.
   # See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/349

kustomization_user.tf

@@ -1,5 +1,5 @@
 locals {
-  user_kustomization_templates = try(fileset("extra-manifests", "*.yaml.tpl"), toset([]))
+  user_kustomization_templates = try(fileset("extra-manifests", "**/*.yaml.tpl"), toset([]))
 }
 
 resource "null_resource" "kustomization_user" {
@@ -13,6 +13,12 @@ resource "null_resource" "kustomization_user" {
     port           = var.ssh_port
   }
 
+  provisioner "remote-exec" {
+    inline = [
+      "mkdir -p $(dirname /var/user_kustomize/${each.key})"
+    ]
+  }
+
   provisioner "file" {
     content     = templatefile("extra-manifests/${each.key}", var.extra_kustomize_parameters)
     destination = replace("/var/user_kustomize/${each.key}", ".yaml.tpl", ".yaml")
@@ -42,7 +48,7 @@ resource "null_resource" "kustomization_user_deploy" {
   provisioner "remote-exec" {
     # Debugging: "sh -c 'for file in $(find /var/user_kustomize -type f -name \"*.yaml\" | sort -n); do echo \"\n### Template $${file}.tpl after rendering:\" && cat $${file}; done'",
     inline = compact([
-      "rm -f /var/user_kustomize/*.yaml.tpl",
+      "rm -f /var/user_kustomize/**/*.yaml.tpl",
       "echo 'Applying user kustomization...'",
       "kubectl apply -k /var/user_kustomize/ --wait=true",
       var.extra_kustomize_deployment_commands