scheduler-deploy: push secrets out via encrypted secrets

[josegonzalez]

Deployments currently use bare environment variables. Although secrets are in plain-text on disk for Dokku, access to the Dokku installation can be fairly locked down, so this isn't a huge issue. However, using plaintext in a scheduler context isn't the most secure thing, as the resource manifests can be inspected by anyone with cluster access. We should instead use encrypted secrets as environment variables.

Creating a secret with every environment variable is a fairly trivial. Here is the json we need to generate and apply to when calling config:set and config:unset:

{
  "apiVersion": "v1",
  "kind": "Secret",
  "metadata": {
    "name": "$APP/environment-variables",
    "labels": {
      "app": "$APP"
    }
 },
  "type": "Opaque",
  "stringData": {
    "$KEY": "$VALUE",
    "$KEY2": "$VALUE2"
  }
}

There is also a limit of 1MB for environment variables, but we can live with this limit for now. If users end up going over this limit, we can revisit this problem, but its fairly unlikely that such a thing happens...

Once we do that, we can switch each env var entry to valueFrom.secretKeyRef as shown here:

{
  "name": "$KEY",
  "valueFrom": {
    "secretKeyRef": {
      "name": "$APP/environment-variables",
      "key": "$KEY"
    }
  }
}

[josegonzalez]

If we use data instead of stringData, each value needs to be base64-encoded.

[josegonzalez]

We can actually use envFrom.secretRef to fetch an env from all the keys in a single variable, so thats cool: https://kubernetes.io/docs/concepts/configuration/secret/#use-case-as-container-environment-variables

This is as of 1.16+, so might not be great to switch to immediately, but I believe we can create the correct env entries otherwise.