Cannot reinstall/upgrade elemental-operator after a Rancher Manager migration

[ldevulder]

I deployed Rancher Manager (Stable, v2.9.3) with latest elemental-operator (Dev version) and backup-restore operator and I encountered some issues.

First one is that metadata Kind is not saved, I opened a PR for this.

Second one is that after migrating/reinstalling the whole Rancher Manager from the backup I'm not able to reinstall/upgrade elemental-operator anymore with helm (CRDs chart is OK).
I have this error:

$ helm upgrade --install --devel elemental-operator \
      oci://registry.opensuse.org/isv/rancher/elemental/dev/charts/rancher/elemental-operator-chart \
      --namespace cattle-elemental-system \
      --create-namespace --wait --wait-for-jobs
Release "elemental-operator" does not exist. Installing it now.
Pulled: registry.opensuse.org/isv/rancher/elemental/dev/charts/rancher/elemental-operator-chart:1.8.0-dev-378.2
Digest: sha256:3930a3682c33f137b63633d5a53ba1639f6dd7da5093b426b1a848a9eec93f4a
W1031 18:02:56.474101   18035 warnings.go:70] unknown field "roleRef.namespace"
Error: cannot patch "sl-micro-6.0-baremetal" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.0-baremetal" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sl-micro-6.0-base" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.0-base" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sl-micro-6.0-kvm" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.0-kvm" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sle-micro-6.0-rt" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sle-micro-6.0-rt" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "unstable-testing-channel" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "unstable-testing-channel" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update
Error while deploying elemental-operator!

Please note that after the migration/reinstallation elemental-operator seems to work as expected, only the upgrade/reinstallation with helm is not working.

Maybe something is still missing in the backup of elemental-operator, but I'm not able to figure out what...

[davidcassany]

Gonna have a closer look next week. To me this smells as an issue related to the dynamic channels provided by helm 🤔 My bet is that this logic might not be fully compatible with the current backup and restore process:

elemental-operator/.obs/chartfile/elemental-operator-helm/templates/channels.yaml

Lines 33 to 46 in f300a43

	# Keep pre-existing channels managed by Helm if they do not match with the current default
	# this way if an upgrade introduces a new channel any pre-existing channel managed by Helm is not deleted
	{{ range $index, $channel := (lookup "elemental.cattle.io/v1beta1" "ManagedOSVersionChannel" "fleet-default" "").items }}
	{{ if and (eq (index $channel.metadata.labels "app.kubernetes.io/managed-by") "Helm") (ne $channel.metadata.name $defChannelName) }}
	---
	apiVersion: elemental.cattle.io/v1beta1
	kind: ManagedOSVersionChannel
	metadata:
	name: {{ $channel.metadata.name }}
	namespace: fleet-default
	spec:
	{{- toYaml $channel.spec | nindent 2}}
	{{ end }}
	{{ end }}

[ldevulder]

I saw that the OS channels and the OS are created in namespace fleet-default and Metadata is in cattle-element-system, could this have an impact?

[davidcassany]

Please note that after the migration/reinstallation elemental-operator seems to work as expected, only the upgrade/reinstallation with helm is not working.

@ldevulder could you relate give me some further details steps of the use case that is functional vs the one that is not? I assume installing clean rancher and elemental, then making a backup and then migrating it to another clean install seams to work. But it doesn't work if the operator was upgraded before making the backup, is that correct?

[davidcassany]

I am also wondering if this will be solved by moving dev to 6.1 channels, probably this is also caused by changing some channel data in dev for 6.0 channels, which shouldn't be there in any case.

[ldevulder]

@davidcassany I tested an upgrade from Stable operator to Dev one after a restore and it worked:

Release "elemental-operator" does not exist. Installing it now.
Pulled: registry.opensuse.org/isv/rancher/elemental/dev/charts/rancher/elemental-operator-chart:1.8.0-dev-380.1
Digest: sha256:259d888269e53758c1d63360904f51e22e8b7dee92318dd619e3094a6abd6590
W1112 12:45:28.407818   19392 warnings.go:70] unknown field "roleRef.namespace"
NAME: elemental-operator
LAST DEPLOYED: Tue Nov 12 12:45:27 2024
NAMESPACE: cattle-elemental-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

The only remaining issue is this warning:

W1112 12:45:28.407818   19392 warnings.go:70] unknown field "roleRef.namespace"

I also still have the issue when I try to reinstall the Stable version of the operator, but sounds logical as nothing has been changed on that side.

[ldevulder]

@davidcassany I just tested to reinstall the Dev version of the operator after a restore with a previously Dev operator installed and I have the same error:

Release "elemental-operator" does not exist. Installing it now.
Pulled: registry.opensuse.org/isv/rancher/elemental/dev/charts/rancher/elemental-operator-chart:1.8.0-dev-380.1
Digest: sha256:259d888269e53758c1d63360904f51e22e8b7dee92318dd619e3094a6abd6590
W1113 15:58:15.672527   32592 warnings.go:70] unknown field "roleRef.namespace"
Error: cannot patch "sl-micro-6.1-baremetal" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.1-baremetal" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sl-micro-6.1-base" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.1-base" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sl-micro-6.1-kvm" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sl-micro-6.1-kvm" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "sle-micro-6.1-rt" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "sle-micro-6.1-rt" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "unstable-testing-channel" with kind ManagedOSVersionChannel: managedosversionchannels.elemental.cattle.io "unstable-testing-channel" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

So an upgrade from Stable to Dev works but not a re-installation.