supporting factory reset

[cgwalters]

This is an extension of coreos/fedora-coreos-tracker#399

I think at a basic level, to "light reprovision" a node it should work to do:

$ oc debug node $x -- /bin/bash -c 'mount -o remount,rw /host/boot && touch /host/boot/ignition.firstboot'

Then, oc edit node/$x and forcibly update the currentConfig+desiredConfig annotations to match the current rendered config value for the pool.

Then, oc debug node $x -- chroot /host reboot

[cgwalters]

xref https://github.com/openshift/machine-config-operator/blob/master/docs/FAQ.md#q-can-i-use-the-mco-to-re-partition-or-re-install

[cgwalters]

OK a lot of internal chat about this , the desire is to try to un-wedge e.g. a control plane node if a bad MC rolled out. But it clearly generalizes to workers too.

So my strawman is:

• Add /usr/libexec/machine-config-daemon reset-config written in Go that does the above touch plus also reconciles the annotations on the node
• Change the default templates to inject /usr/local/bin/openshift-node-reset that is just an exec /run/bin/machine-config-daemon reset-config (but allows us to change the implementation in the future)
• Add CI coverage for this in the MCO repo

Now...hmm actually /run/bin will (intentionally) go away on reboot, the idea is we want to sync the MCD's binary. Perhaps instead we should dynamically write /usr/local/bin/machine-config-daemon (persistent) but update it based on contents from the pod when that runs.

[cgwalters]

And another further thing we can do once we have this fuctionality is add support for cluster initated resets, like the admin adds an annotation io.openshift.machineconfig.reset="true" or whatever to the node object, and the MCD says "OK" and does this for you.

[deads2k]

So my strawman is:

• Add /usr/libexec/machine-config-daemon reset-config written in Go that does the above touch plus also reconciles the annotations on the node
• Change the default templates to inject /usr/local/bin/openshift-node-reset that is just an exec /run/bin/machine-config-daemon reset-config (but allows us to change the implementation in the future)
• Add CI coverage for this in the MCO repo

I like the idea. I think it covers the three different failures we've seen on actual clusters that I think we can resolve

1. without a running kubelet
2. without a running crio
3. without the ability to pull images

I don't think we can handle the without a functional network in any reasonable way.

Trying to get creds for for your step 1 is a thing I think @sttts can work out on masters

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.