Merge pull request #191 from jacobweinstock/capt-playground

Add CAPT playground:

## Description

<!--- Please describe what this PR is going to change -->
This will allow users to try out CAPT. The CAPT playground uses [Task](https://taskfile.dev/) and a few bash scripts to create the playground.
## Why is this needed

<!--- Link to issue you have raised -->

Fixes: #165 

## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->


## How are existing users impacted? What migration steps/scripts do we need?

<!--- Fixes a bug, unblocks installation, removes a component of the stack etc -->
<!--- Requires a DB migration script, etc. -->
I've tested this locally on a few different machines.

## Checklist:

I have:

- [x] updated the documentation and/or roadmap (if required)
- [ ] added unit or e2e tests
- [ ] provided instructions on how to upgrade

.github/workflows/ci-non-go.sh

@@ -18,7 +18,7 @@ if ! shfmt -f . | xargs shfmt -s -l -d; then
 	failed=1
 fi
 
-if ! rufo vagrant/Vagrantfile; then
+if ! rufo stack/vagrant/Vagrantfile; then
 	failed=1
 fi
 

.gitignore

@@ -1 +1,6 @@
-.vagrant
+.vagrant
+error.log
+.task
+.state
+capt/output/
+.vscode/

README.md

@@ -1,40 +1,7 @@
 # Playground
 
-The playground is an example deployment of the Tinkerbell stack for use in learning and testing. It is not a production reference architecture.
-Please use the [Helm chart](https://github.com/tinkerbell/charts) for production deployments.
+Welcome to the Tinkerbell Playground! This playground repository holds example deployments for use in learning and testing.
+The following playgrounds are available:
 
-## Quick-Starts
-
-The following quick-start guides will walk you through standing up the Tinkerbell stack.
-There are a few options for this.
-Pick the one that works best for you.
-
-## Options
-
-- [Vagrant and VirtualBox](docs/quickstarts/VAGRANTVBOX.md)
-- [Vagrant and Libvirt](docs/quickstarts/VAGRANTLVIRT.md)
-- [Kubernetes](docs/quickstarts/KUBERNETES.md)
-
-## Next Steps
-
-By default the Vagrant quickstart guides automatically install Ubuntu on the VM (machine1). You can provide your own OS template. To do this:
-
-1. Login to the stack VM
-
-   ```bash
-   vagrant ssh stack
-   ```
-
-1. Add your template. An example Template object can be found [here](https://github.com/tinkerbell/tink/tree/main/config/crd/examples/template.yaml) and more Template documentation can be found [here](https://tinkerbell.org/docs/concepts/templates/).
-
-   ```bash
-   kubectl apply -f my-OS-template.yaml
-   ```
-
-1. Create the workflow. An example Workflow object can be found [here](https://github.com/tinkerbell/tink/tree/main/config/crd/examples/workflow.yaml).
-
-   ```bash
-   kubectl apply -f my-custom-workflow.yaml
-   ```
-
-1. Restart the machine to provision (if using the vagrant playground test machine this is done by running `vagrant destroy -f machine1 && vagrant up machine1`)
+- [Tinkerbell stack playground](stack/README.md)
+- [Cluster API Provider Tinkerbell (CAPT) playground](capt/README.md)

capt/README.md

@@ -0,0 +1,87 @@
+# Cluster API Provider Tinkerbell (CAPT) Playground
+
+The Cluster API Provider Tinkerbell (CAPT) is a Kubernetes Cluster API provider that uses Tinkerbell to provision machines. You can find more information about CAPT [here](https://github.com/tinkerbell/cluster-api-provider-tinkerbell). The CAPT playground is an example deployment for use in learning and testing. It is not a production reference architecture.
+
+## Getting Started
+
+The CAPT playground is a tool that will create a local CAPT deployment and a single workload cluster. This includes creating and installing a Kubernetes cluster (KinD), the Tinkerbell stack, all CAPI and CAPT components, Virtual machines that will be used to create the workload cluster, and a Virtual BMC server to manage the VMs.
+
+Start by reviewing and installing the [prerequisites](#prerequisites) and understanding and customizing the [configuration file](./config.yaml) as needed.
+
+## Prerequisites
+
+### Binaries
+
+- [Libvirtd](https://wiki.debian.org/KVM) >= libvirtd (libvirt) 8.0.0
+- [Docker](https://docs.docker.com/engine/install/) >= 24.0.7
+- [Helm](https://helm.sh/docs/intro/install/) >= v3.13.1
+- [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) >= v0.20.0
+- [clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start#install-clusterctl) >= v1.6.0
+- [kubectl](https://www.downloadkubernetes.com/) >= v1.28.2
+- [virt-install](https://virt-manager.org/) >= 4.0.0
+- [task](https://taskfile.dev/installation/) >= 3.37.2
+
+### Hardware
+
+- at least 60GB of free and very fast disk space (etcd is very disk I/O sensitive)
+- at least 8GB of free RAM
+- at least 4 CPU cores
+
+## Usage
+
+Start by looking at the [`config.yaml`](./config.yaml) file. This file contains the configuration for the playground. You can customize the playground by changing the values in this file. We recommend you start with the defaults to get familiar with the playground before customizing.
+
+Create the CAPT playground:
+
+```bash
+# Run the creation process and follow the outputted next steps at the end of the process.
+task create-playground
+```
+
+Delete the CAPT playground:
+
+```bash
+task delete-playground
+```
+
+## Next Steps
+
+With the playground up and running and a workload cluster created, you can run through a few CAPI lifecycle operations.
+
+### Move/pivot the Tinkerbell stack and CAPI/CAPT components to a workload cluster
+
+To be written.
+
+### Upgrade the management cluster
+
+To be written.
+
+### Upgrade the workload cluster
+
+To be written.
+
+### Scale out the workload cluster
+
+To be written.
+
+### Scale in the workload cluster
+
+To be written.
+
+## Known Issues
+
+### DNS issue
+
+KinD on Ubuntu has a known issue with DNS resolution in KinD pod containers. This affect the Download of HookOS in the Tink stack helm deployment. There are a few [known workarounds](https://github.com/kubernetes-sigs/kind/issues/1594#issuecomment-629509450). The recommendation for the CAPT playground is to add a DNS nameservers to Docker's `daemon.json` file. This can be done by adding the following to `/etc/docker/daemon.json`:
+
+```json
+{
+  "dns": ["1.1.1.1"]
+}
+```
+
+Then restart Docker:
+
+```bash
+sudo systemctl restart docker
+```

capt/Taskfile.yaml

@@ -0,0 +1,128 @@
+version: "3"
+
+includes:
+  create: ./tasks/Taskfile-create.yaml
+  delete: ./tasks/Taskfile-delete.yaml
+  vbmc: ./tasks/Taskfile-vbmc.yaml
+  capi: ./tasks/Taskfile-capi.yaml
+
+vars:
+  OUTPUT_DIR:
+    sh: echo $(yq eval '.outputDir' config.yaml)
+  CURR_DIR:
+    sh: pwd
+  STATE_FILE: ".state"
+  STATE_FILE_FQ_PATH:
+    sh: echo {{joinPath .CURR_DIR .STATE_FILE}}
+
+tasks:
+  create-playground:
+    silent: true
+    summary: |
+      Create the CAPT playground. Use the .playground file to define things like cluster size and Kubernetes version.
+    cmds:
+      - task: system-deps-warnings
+      - task: validate-binaries
+      - task: ensure-output-dir
+      - task: generate-state
+      - task: create:playground-ordered
+      - task: next-steps
+
+  delete-playground:
+    silent: true
+    summary: |
+      Delete the CAPT playground.
+    cmds:
+      - task: validate-binaries
+      - task: delete:playground
+
+  validate-binaries:
+    silent: true
+    summary: |
+      Validate all required dependencies for the CAPT playground.
+    cmds:
+      - for:
+          [
+            "virsh",
+            "docker",
+            "helm",
+            "kind",
+            "kubectl",
+            "clusterctl",
+            "virt-install",
+            "brctl",
+            "yq",
+          ]
+        cmd: command -v {{ .ITEM }} >/dev/null || echo "'{{ .ITEM }}' was not found in the \$PATH, please ensure it is installed."
+        # sudo apt install virtinst # for virt-install
+        # sudo apt install bridge-utils # for brctl
+
+  system-deps-warnings:
+    summary: |
+      Run CAPT playground system warnings.
+    silent: true
+    cmds:
+      - echo "Please ensure you have the following:"
+      - echo "60GB of free and very fast disk space (etcd is very disk I/O sensitive)"
+      - echo "8GB of free RAM"
+      - echo "4 CPU cores"
+
+  ensure-output-dir:
+    summary: |
+      Create the output directory.
+    cmds:
+      - mkdir -p {{.OUTPUT_DIR}}
+      - mkdir -p {{.OUTPUT_DIR}}/xdg
+    status:
+      - echo ;[ -d {{.OUTPUT_DIR}} ]
+      - echo ;[ -d {{.OUTPUT_DIR}}/xdg ]
+
+  generate-state:
+    summary: |
+      Populate the state file.
+    sources:
+      - config.yaml
+    generates:
+      - .state
+    cmds:
+      - ./scripts/generate_state.sh config.yaml .state
+
+  next-steps:
+    silent: true
+    summary: |
+      Next steps after creating the CAPT playground.
+    vars:
+      NAMESPACE:
+        sh: yq eval '.namespace' {{.STATE_FILE_FQ_PATH}}
+      NODE_BASE:
+        sh: yq eval '.vm.baseName' {{.STATE_FILE_FQ_PATH}}
+      CLUSTER_NAME:
+        sh: yq eval '.clusterName' {{.STATE_FILE_FQ_PATH}}
+      KIND_KUBECONFIG:
+        sh: yq eval '.kind.kubeconfig' {{.STATE_FILE_FQ_PATH}}
+    cmds:
+      - |
+        echo
+        echo The workload cluster is now being created.
+        echo Once the cluster nodes are up and running, you will need to deploy a CNI for the cluster to be fully functional.
+        echo The management cluster kubeconfig is located at: {{.KIND_KUBECONFIG}}
+        echo The workload cluster kubeconfig is located at: {{.OUTPUT_DIR}}/{{.CLUSTER_NAME}}.kubeconfig
+        echo
+        echo 1. Watch and wait for the first control plane node to be provisioned successfully: STATE_SUCCESS
+        echo "KUBECONFIG={{.KIND_KUBECONFIG}} kubectl get workflows -n {{.NAMESPACE}} -w"
+        echo
+        echo
+        echo 2. Watch and wait for the Kubernetes API server to be ready and responding:
+        echo "until KUBECONFIG={{.OUTPUT_DIR}}/{{.CLUSTER_NAME}}.kubeconfig kubectl get node; do echo 'Waiting for Kube API server to respond...'; sleep 5; done"
+        echo
+        echo 3. Deploy a CNI
+        echo Cilium
+        echo "KUBECONFIG={{.OUTPUT_DIR}}/{{.CLUSTER_NAME}}.kubeconfig cilium install"
+        echo or KUBEROUTER
+        echo "KUBECONFIG={{.OUTPUT_DIR}}/{{.CLUSTER_NAME}}.kubeconfig kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml"
+        echo
+        echo 4. Watch and wait for all nodes to join the cluster and be ready:
+        echo "KUBECONFIG={{.OUTPUT_DIR}}/{{.CLUSTER_NAME}}.kubeconfig kubectl get nodes -w"
+      - touch {{.OUTPUT_DIR}}/.next-steps-displayed
+    status:
+      - echo ;[ -f {{.OUTPUT_DIR}}/.next-steps-displayed ]

capt/config.yaml

@@ -0,0 +1,29 @@
+---
+clusterName: "capt-playground"
+outputDir: "output"
+namespace: "tink"
+counts:
+  controlPlanes: 1
+  workers: 1
+  spares: 3
+versions:
+  capt: 0.5.3
+  chart: 0.4.5
+  kube: v1.28.3
+  os: 20.04
+  kubevip: 0.8.0
+os:
+  registry: ghcr.io/jacobweinstock/capi-images
+  distro: ubuntu
+  sshKey: ""
+vm:
+  baseName: "node"
+  cpusPerVM: 2
+  memInMBPerVM: 2048
+  diskSizeInGBPerVM: 10
+  diskPath: "/tmp"
+virtualBMC:
+  containerName: "virtualbmc"
+  image: ghcr.io/jacobweinstock/virtualbmc
+  user: "root"
+  pass: "calvin"

capt/scripts/create_vms.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+set -euo pipefail
+
+# Create VMs
+
+function main() {
+	declare -r STATE_FILE="$1"
+	declare -r OUTPUT_DIR=$(yq eval '.outputDir' "$STATE_FILE")
+	declare BRIDGE_NAME="$(yq eval '.kind.bridgeName' "$STATE_FILE")"
+	declare CPUS="$(yq eval '.vm.cpusPerVM' "$STATE_FILE")"
+	declare MEM="$(yq eval '.vm.memInMBPerVM' "$STATE_FILE")"
+	declare DISK_SIZE="$(yq eval '.vm.diskSizeInGBPerVM' "$STATE_FILE")"
+	declare DISK_PATH="$(yq eval '.vm.diskPath' "$STATE_FILE")"
+
+	while IFS=$',' read -r name mac; do
+		# create the VM
+		virt-install \
+			--description "CAPT VM" \
+			--ram "$MEM" --vcpus "$CPUS" \
+			--os-variant "ubuntu20.04" \
+			--graphics "vnc" \
+			--boot "uefi,firmware.feature0.name=enrolled-keys,firmware.feature0.enabled=no,firmware.feature1.name=secure-boot,firmware.feature1.enabled=yes" \
+			--noautoconsole \
+			--noreboot \
+			--import \
+			--connect "qemu:///system" \
+			--name "$name" \
+			--disk "path=$DISK_PATH/$name-disk.img,bus=virtio,size=10,sparse=yes" \
+			--network "bridge:$BRIDGE_NAME,mac=$mac"
+	done < <(yq e '.vm.details.[] | [key, .mac] | @csv' "$STATE_FILE")
+}
+
+main "$@"

capt/scripts/generate_bmc.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+set -euo pipefail
+
+# This script creates the BMC machine yaml files needed for the CAPT playground.
+
+function main() {
+	declare -r STATE_FILE="$1"
+	declare -r OUTPUT_DIR=$(yq eval '.outputDir' "$STATE_FILE")
+
+	rm -f "$OUTPUT_DIR"/bmc-machine*.yaml
+
+	namespace=$(yq eval '.namespace' "$STATE_FILE")
+	bmc_ip=$(yq eval '.virtualBMC.ip' "$STATE_FILE")
+
+	while IFS=$',' read -r name port; do
+		export NODE_NAME="$name"
+		export BMC_IP="$bmc_ip"
+		export BMC_PORT="$port"
+		export NAMESPACE="$namespace"
+		envsubst "$(printf '${%s} ' $(env | cut -d'=' -f1))" <templates/bmc-machine.tmpl >"$OUTPUT_DIR"/bmc-machine-"$NODE_NAME".yaml
+		unset NODE_NAME
+		unset BMC_IP
+		unset BMC_PORT
+		unset NAMESPACE
+	done < <(yq e '.vm.details.[] | [key, .bmc.port] | @csv' "$STATE_FILE")
+}
+
+main "$@"