k8s docker-compose (tinkerbell#154)

## Description


This PR brings up the sandbox via Docker compose using the Kubernetes backend for all service. This does not completely remove the postgres backend setup but moves all the compose with postgres into an isolated directory (deploy/compose/postgres) that can be removed when we're ready. 

> I did not touch the terraform setup. I need some help validating that one. please and thank you. CC @mmlb @displague

## Why is this needed



Fixes: tinkerbell#142 tinkerbell#45 tinkerbell#118 tinkerbell#131 tinkerbell#133 tinkerbell#145 tinkerbell#148 
- This "fixes" a quite a few issues related to TLS cert generation. This is the case because we are not using TLS in this deployment. Also see, tinkerbell/tink#555.
- This also "fixes" any issues related to the internal registry as that is removed as the default.

## How Has This Been Tested?



Manually tested vagrant with virtualbox (on a Mac), vagrant with libvirt (on Ubuntu 22.04), and docker-compose (on on Ubuntu 22.04). 


## How are existing users impacted? What migration steps/scripts do we need?
There is no migration support. Users will need to follow a quick start guide to get started.





## Checklist:

I have:

- [x] updated the documentation and/or roadmap (if required)
- [ ] added unit or e2e tests
- [ ] provided instructions on how to upgrade

.gitignore

@@ -1,8 +1,9 @@
 # hidden files/dirs
 .*
-!deploy/compose/state/webroot/misc/osie/current/.keep
-!deploy/compose/state/webroot/workflow/.keep
+!deploy/compose/postgres/state/webroot/misc/osie/current/.keep
+!deploy/compose/postgres/state/webroot/workflow/.keep
 !deploy/.env
+!deploy/compose/.env
 !.gitignore
 !.github/
 
@@ -16,9 +17,12 @@
 
 compose.tar.gz
 compose.zip
-deploy/compose/state/webroot/*.gz
-deploy/compose/state/webroot/misc/osie/current/*
-deploy/compose/state/webroot/workflow/*
+deploy/compose/postgres/state/webroot/*.gz
+deploy/compose/postgres/state/webroot/misc/osie/current/*
+deploy/compose/postgres/state/webroot/workflow/*
+deploy/compose/manifests/manifests.yaml
+deploy/compose/state/*
+!deploy/compose/state/.keep
 envrc
 out
 workflow_id.txt

README.md

@@ -20,28 +20,23 @@ Tinkerbell.org has a [list of guides](https://docs.tinkerbell.org/deploying-oper
 You can also create your own.
 The following docs will help you get started.
 
-1. [Create Hardware Data](https://docs.tinkerbell.org/setup/local-vagrant/#creating-the-workers-hardware-data)
-2. [Create a Template](https://docs.tinkerbell.org/setup/local-vagrant/#creating-a-template)
-3. [Create a Workflow](https://docs.tinkerbell.org/setup/local-vagrant/#creating-the-workflow)
+1. [Example Hardware object](https://github.com/tinkerbell/tink/tree/main/config/crd/examples/hardware.yaml)
+2. [Example Template object](https://github.com/tinkerbell/tink/tree/main/config/crd/examples/template.yaml)
+   - Template [documentation](https://docs.tinkerbell.org/templates/)
+3. [Example Workflow object](https://github.com/tinkerbell/tink/tree/main/config/crd/examples/workflow.yaml)
 
 ### In the Sandbox
 
-1. Create your own templates
+1. Add your templates
 
    ```bash
-   docker exec -i compose_tink-cli_1 tink template create < ./custom-template.yaml
+   kubectl apply -f my-custom-template.yaml
    ```
 
-2. Upload any container images you want to use in the templates to the internal registry
+2. Create the workflow
 
    ```bash
-   docker run -it --rm quay.io/containers/skopeo copy --all --dest-tls-verify=false --dest-creds="admin":"Admin1234" docker://hello-world docker://192.168.56.4/hello-world
+   kubectl apply -f my-custom-workflow.yaml
    ```
 
-3. Create a workflow
-
-   ```bash
-   docker exec -i compose_tink-cli_1 tink workflow create -t <TEMPLATE ID> -r '{"device_1":"08:00:27:00:00:01"}')
-   ```
-
-4. Restart the machine to provision (if using the vagrant sandbox test machine this is done by running `vagrant destroy -f machine1 && vagrant up machine1`)
+3. Restart the machine to provision (if using the vagrant sandbox test machine this is done by running `vagrant destroy -f machine1 && vagrant up machine1`)

deploy/compose/.env

@@ -1,15 +1,5 @@
-# These must be defined above/before first use.
-# Use of these variables *must* be in ${} form, otherwise docker-compose won't substitute when processing this file
-vOSIE=v0.7.0
-vTINK=sha-16186501
-
-# Probably don't want to mess with these, unless you know you do
-FACILITY=onprem
-TINKERBELL_REGISTRY_PASSWORD=Admin1234
-TINKERBELL_REGISTRY_USERNAME=admin
-TINKERBELL_TLS= false
-
 # Can be set to your own hook builds
+vOSIE=v0.7.0
 OSIE_DOWNLOAD_URLS=https://github.com/tinkerbell/hook/releases/download/${vOSIE}/hook_x86_64.tar.gz,https://github.com/tinkerbell/hook/releases/download/${vOSIE}/hook_aarch64.tar.gz
 
 TINKERBELL_HARDWARE_MANIFEST=/manifests/hardware/hardware.json
@@ -19,9 +9,16 @@ TINKERBELL_CLIENT_IP=192.168.56.43
 TINKERBELL_CLIENT_MAC=08:00:27:9e:f5:3a
 TINKERBELL_HOST_IP=192.168.56.4
 
-# Images used by docker-compose natively or in terraform/vagrant, update if necessary
-BOOTS_IMAGE=quay.io/tinkerbell/boots:sha-505785d7
-HEGEL_IMAGE=quay.io/tinkerbell/hegel:sha-592588cf
-TINK_CLI_IMAGE=quay.io/tinkerbell/tink-cli:${vTINK}
-TINK_SERVER_IMAGE=quay.io/tinkerbell/tink:${vTINK}
-TINK_WORKER_IMAGE=quay.io/tinkerbell/tink-worker:${vTINK}
+# Images used by docker compose natively or in terraform/vagrant, update if necessary
+BOOTS_IMAGE=quay.io/tinkerbell/boots:v0.7.0
+HEGEL_IMAGE=quay.io/tinkerbell/hegel:v0.8.0
+TINK_VERSION=v0.8.0
+TINK_SERVER_IMAGE=quay.io/tinkerbell/tink:${TINK_VERSION}
+TINK_CONTROLLER_IMAGE=quay.io/tinkerbell/tink-controller:${TINK_VERSION}
+TINK_WORKER_IMAGE=quay.io/tinkerbell/tink-worker:${TINK_VERSION}
+RUFIO_VERSION=v0.1.0
+RUFIO_IMAGE=quay.io/tinkerbell/rufio:${RUFIO_VERSION}
+K3S_IMAGE=rancher/k3s:v1.24.4-k3s1
+
+# DISK_DEVICE is used in the manifests/hardware.yaml
+DISK_DEVICE=/dev/sda