WIP Add MCO Flattened Ignition proposal

enhancements/machine-config/mco-flattened-ignition.md

@@ -0,0 +1,175 @@
+---
+title: MCO Support flattened ignition config
+authors:
+  - "@hardys"
+reviewers:
+  - "@celebdor"
+  - "@cgwalters"
+  - "@crawford"
+  - "@kirankt"
+  - "@runcom"
+  - "@stbenjam"
+approvers:
+  - "@cgwalters"
+  - "@crawford"
+  - "@runcom"
+
+creation-date: 2020-08-21
+last-updated: 2020-09-02
+status: **provisional**|implementable|implemented|deferred|rejected|withdrawn|replaced
+---
+
+# MCO Support flattened ignition config
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Summary
+
+This enhancement proposes adding an interface to the MCO such that a flattened
+ignition config, that is a combination of the managed pointer config and the
+MCO rendered config (inlined via a [data URL](https://tools.ietf.org/html/rfc2397)).
+
+## Motivation
+
+In some situations it is necessary to perform network configuration before it
+is possible to download the rendered config from the MCS in the ramdisk where
+ignition evaluates the pointer config.
+
+This leads to a chicken-egg problem, where a user could configure the network
+via ignition, but ignition cannot apply any config until it has downloaded
+any append/merge directives.
+
+This situation is quite common in baremetal environments, e.g a bonded pair of nics
+(where the native vlan is often used for PXE provisioning),
+with the controlplane network configured via a VLAN on the same physical interface.
+
+For UPI deployments, it may be possible to work around this situation when
+using coreos-install, since this has a `--copy-network` option which can drop
+NetworkManager config into /boot, but it would be a more consistent user
+experience if all configuration was possible via ignition?
+
+For baremetal IPI deployments currently the only workaround is to customize the
+OS image, since the deploy toolchain currently does not use coreos-install,
+or include any interface to drop additional files into /boot during deployment.
+
+### Goals
+
+ * Provide a solution to customer requirements for baremetal IPI around bond/VLAN
+   and other configurations for controlplane networking
+ * Avoid a platform-specific approach, which could be fragile and/or hard to maintain
+
+### Non-Goals
+
+The discussion of interfaces here only considers the existing interfaces, not
+proposals around a future [declarative network configuration](https://github.com/openshift/enhancements/pull/399)
+although we need to ensure the approach taken in each case is aligned.
+
+### User Stories
+
+#### Story 1
+
+As a UPI user, I would like to perform network configuration in the same
+way as other customization, e.g via MachineConfig manifests or ignition.
+
+#### Story 2
+
+As a baremetal IPI user, I need to deploy in an environment where network
+configuration is required before access to the controlplane network is possible.
+
+## Design Details
+
+The MCO currently maintains two ignition artifacts, a managed instance of the
+pointer ignition config, stored in a Secret, and the rendered config stored
+in a MachineConfig object (both per role, not per machine).
+
+Both artifacts are accessible via the MCS (although not currently from within
+the cluster ref https://github.com/openshift/machine-config-operator/issues/1690)
+so they may be consumed during deployment.
+
+The proposal is to create a third managed artifact, which is the flattened
+combination of the pointer and rendered ignition configs, so that when
+required it may be consumed instead of requiring runtime download of the rendered
+config via the pointer URL.
+
+Open Questions for discussion:
+  * Should the new artifact be another MachineConfig or a Secret?
+  * How do we resolve the issue of CAPBM not being able to access the MCS,
+    can we reliably determine the MCO managed resource and reference it directly?
+  * Does this approach fit with discussions around declarative network
+    configuration (for day-1 and day-2)?
+
+### Test Plan
+
+**Note:** *Section not required until targeted at a release.*
+
+### Graduation Criteria
+
+**Note:** *Section not required until targeted at a release.*
+
+### Upgrade / Downgrade Strategy
+
+### Version Skew Strategy
+
+## Alternatives
+
+### Native ignition support for early-network config
+
+This was initially proposed via [an ignition issue](https://github.com/coreos/ignition/issues/979)
+where we had some good discussion, but ultimately IIUC the idea of adding a new
+interface for early network configuration to ignition was rejected.
+
+### Implement config flattening in platform-specific repos
+
+The ignition configuration is consumed at two points during deployment for IPI baremetal
+deployments:
+
+  * Via terraform (when the installer deploys the controlplane
+    nodes using [terraform-provider-ironic](https://github.com/openshift-metal3/terraform-provider-ironic).
+  * Workers are then deployed via the machine-api, using a
+    [baremetal Cluster API Provider](https://github.com/openshift/cluster-api-provider-baremetal/)
+
+The ironic terraform provider is designed to be generic, and thus we would prefer
+not to add OS specific handling of the user-data there, but it would be technically
+possible to do the flattening in some terraform provider, and duplicate that
+logic in the cluster-api-provider-baremetal.
+
+This option is probably suboptimal though, since it would require duplicate code,
+and knowledge of e.g ignition version/interfaces outside of the MCO managed
+configs.  It also would not be available for any other platforms or UPI usage.
+
+### Add support for injecting NM config to IPI
+
+This might be possible using a recently added [ironic python agent feature](https://specs.openstack.org/openstack/ironic-specs/specs/approved/in-band-deploy-steps.html)
+using that interface, it could be possible to inject network configuration in a
+similar way to coreos-install.
+
+However this is a very bleeding edge feature (not yet documented), and it means
+maintaining a custom interface that would be different to both the proven
+uncustomized OSP deploy ramdisk, and the coreos-deploy toolchain.
+
+### Convert IPI to coreos-install based deployment
+
+Long term, from an OpenShift perspective, this probably makes sense, however
+in the short/medium term, the IPI deployment components are based on Ironic,
+and that is only really tested with the ironic deploy ramdisk (upstream and
+downstream) - we need to investigate how IPI deployments may be adapted
+to converge with the coreos-install based workflow, but this is likely to
+require considerable planning and development effort to achieve, thus is
+not implementable as an immediate solution.
+
+### Customize images
+
+Currently the only option for IPI based deployments is to customize the
+OS image, then provide a locally cached copy of this custom image to the
+installer via the `clusterOSImage` install-config option.
+
+This is only a stopgap solution, it is inconvenient for the user, and
+inflexible as it will only work if the configuration required is common
+to every node in the cluster (or for multiple worker machinepools, you
+would require a custom image per pool).