Figure out how to support switching storage type after workspace creation

[amisevsk]

Description

Currently, the DevWorkspace Operator uses webhooks to block changing the storage-type of workspaces after they have been created unless we know that doing so won't result in losing changes in the DevWorkspace. For example:

• If a workspace is switching from ephemeral storage to a persistent storage type (e.g. common), the change is allowed since an ephemeral workspace will lose all changes when it is restarted.
• If a workspace uses the shared PVC for storage but hasn't been started yet (does not have the storage finalizer), changes to storage-type are allowed.

However, other changes are not permitted:

• Switching from common storage, once the storage finalizer has been set, is blocked, as this would result all data stored in the common PVC to be orphaned (since the workspace no longer uses the common PVC, when it is deleted files in the common PVC will be left in-place).
• Changing the default storage type (the storage-type used when no attribute is defined) in DWO is not possible, since switching from e.g. common to ephemeral would leave all existing DevWorkspace files on PVCs without cleanup.

There are two main reasons these restrictions are in place:

1. We want to avoid configuration changes that result in DevWorkspace users losing changes made to their workspace's PVC (e.g. if I'm preparing a PR in a Theia workspace, switching from common would lose any work I've done if it's not pushed to a remote git repository)
2. We really want to avoid orphaning data in PVCs across the cluster, as not fully cleaning up workspaces on deletion could eventually result in common PVCs filling up with untracked data, preventing new workspaces from functioning correctly (while wasting cluster resources)

If we can work around the second issue, we should consider allowing the storage-type to be changed at-will, with the caveat that it may result in loss of data for users. Some potential options to look into:

• Use webhooks to track storage-type history (in annotations or in attributes), so that the correct cleanup can be run when a workspace is deleted.
• Use separate finalizers for each storage type so that cleanup is not linked to the current storage class (e.g. if a workspace has the commonpvc.storage.controller.devfile.io finalizer, run the common PVC cleanup job regardless of current storage-type). This would be a decently-sized refactor but would allow for changing the default storage type as well.

Additional context

We need to make sure cleanup works correctly regardless of the state of workspaces/PVCs -- with the suggested changes, we open the possibility of strange edgecases:

• When the last workspace that uses the common PVC is deleted, the common PVC is deleted instead of being cleaned up by the cleanup job. With the changes, we could switch all workspaces to e.g. ephemeral, preventing the common PVC from being cleaned up
• It's possible for a workspace to have the common PVC finalizer set, but for the PVC to not contain any data for that workspace:
  1. Create common PVC workspace, start it, and switch storage-type to ephemeral
  2. Create and delete another common PVC workspace, resulting in the shared PVC being deleted (there are no common PVC workspaces in the namespace)
  3. Create another common PVC workspace, creating a new common PVC
  4. Delete the original workspace -- it will attempt to clean up the common PVC even though it's been recreated since this workspace used it last.
• We could run into a similar situation to Errored workspaces aren't removed after PVC cleanup failure #873 -- it's possible to delete a workspace that requires the common PVC cleanup job to be run but for the common PVC to no longer exist, and this needs to not result in an error state.
• Since DWO supports existing PVCs (e.g. the claim-che-workspace PVC), we need to make sure all of the above works correctly when that PVC is present as well.