Support a provisioning token for the Machine Config Server

See openshift/machine-config-operator#784

The Ignition configuration can contain secrets, and we want to avoid it being accessible both inside and outside the cluster.

enhancements/machine-config/auth-token.md

@@ -0,0 +1,124 @@
+---
+title: mco-auth-token
+authors:
+  - "@cgwalters"
+reviewers:
+  - "@crawford"
+approvers:
+  - "@ashcrow"
+  - "@crawford"
+  - "@imcleod"
+  - "@runcom"
+creation-date: 2020-08-05
+last-updated: 2020-08-19
+status: provisional
+see-also:
+replaces:
+superseded-by:
+---
+
+# Support a provisioning token for the Machine Config Server
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Summary
+
+This enhancement proposes that for user-provisioned installations we optionally support generating a "provisioning token":
+
+```
+$ dd if=/dev/urandom bs=32 count=1 | base64 > token
+$ oc -n openshift-config create secret generic machineconfig-auth --from-file=token
+```
+
+Installer-provisioned infrastructure environments would implement this automatically.
+
+When configured, requests to the Machine Config Server will require this token to be provided via e.g. `GET /config/worker&token=<value>`.
+
+## Motivation
+
+See https://github.com/openshift/machine-config-operator/pull/784
+
+The Ignition configuration can contain secrets, and we want to avoid it being accessible both inside and outside the cluster.
+
+### Goals
+
+- Increase security
+- Avoid breaking upgrades
+- Easy to implement
+- Easy to understand for operators
+- Eventually support being replaced by something more sophisticated (e.g. TPM2 attestation)
+- Avoid too strong dependence on platform-specific functionality (i.e. UPI AWS should remain close to  UPI bare metal)
+
+### Non-Goals
+
+- Completely rework major parts of the provisioning process today
+
+## Proposal
+
+Change the MachineConfigServer to support requiring a token in order to retrieve the Ignition config.
+
+In IPI/machineAPI managed clusters, the MCO would automatically handle this end to end.
+
+### Implementation Details/Notes/Constraints
+
+The documentation for user provisioned infrastructure would be extended to show how to configure this.  We would also need to debate important details such as whether this is something we want to allow to be configured via `install-config.yaml` or whether it should only be a "day 2" hardening.
+
+For installer-provisioned infrastructure, this will be much easier to implement for installations where the `user-data` provided to nodes is owned by the MCO: https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/user-data-secret-managed.md
+
+In order to avoid race conditions, the MCS might support the "previous" token for a limited period of time (e.g. one hour/day).
+
+#### IPI/machineAPI details
+
+It's probably easiest to start out by automatically generating a rotating secret that works the same way as UPI, and also (per above) support the previous token for a period of time.
+
+### Risks and Mitigations
+
+#### Disaster recovery
+
+See: [Disaster recovery](https://docs.openshift.com/container-platform/4.5/backup_and_restore/disaster_recovery/about-disaster-recovery.html)
+
+This proposes that the secret for provisioning a node is stored in the cluster itself.  But so is the configuration for the cluster, so this is not a new problem.
+
+That said, see "Rotating the token" below.
+
+#### Non-Ignition components fetching the Ignition config
+
+As part of the Ignition spec 3 transition, we needed to deal with non-Ignition consumers of the Ignition config, such as [openshift-ansible](github.com/openshift/openshift-ansible/) and Windows nodes.  These should generally instead fetch the rendered `MachineConfig` object from the cluster, and not access the MCS port via TCP.
+
+#### Rotating the token
+
+We should document rotating the token, though doing so incurs some danger if the administrator forgets to update the pointer configuration.  We should recommend against administrators rotating the token *without* also implementing a "periodic reprovisioning" policy to validate that they can restore workers (and ideally control plane) machines.
+
+Or better, migrate to an IPI/machineAPI managed cluster.
+
+That said, we will need to discuss this in a "node provisioning" section in the documentation, and also link to it from the disaster recovery documentation to ensure that an administrator trying to re-provision a failed control plane node has some documentation if they forgot to update the pointer configuration for the control plane.
+
+This scenario would be easier to debug if [Ignition could report failures to the MCS](https://github.com/coreos/ignition/issues/585).
+
+### Upgrades
+
+This will not apply by default on upgrades, even in IPI/machineAPI managed clusters (to start).  However, an administrator absolutely could enable this "day 2".  Particularly for "static" user provisioned infrastructure where the new nodes are mostly manually provisioned, the burden of adding the token into the process of provisioning would likely be quite low.
+
+## Alternatives
+
+#### Firewalling
+
+Not obvious to system administrators in e.g. bare metal environments and difficult to enforce with 3rd party SDNs.
+
+#### Move all secret data out of Ignition 
+
+This would then gate access on CSR approval, but this introduces bootstrapping problems for the kubelet such as the pull secret.
+
+#### Encrypt secrets in Ignition
+
+Similar to above, we have a bootstrapping problem around fetching the key.
+
+#### Improved node identity
+
+@cgwalters believes that we should support [Trusted Platform Module](https://en.wikipedia.org/wiki/Trusted_Platform_Module)s for the whole provisioning process, including [CSR approval](https://github.com/openshift/cluster-machine-approver).  But this is a much bigger change, and while TPM2 is available in GCP and most modern bare metal, it wouldn't apply in e.g. AWS, and we need a "baseline" solution soon.