Upgrade of pravega cluster is failing intermittently

[anishakj]

Description

Upgrade of pravega cluster is failing intermittently. In operator logs, seeing below output

time="2022-06-23T13:56:25Z" level=info msg="syncing cluster version from 0.10.2-3029.0cbd9f1e9 to 0.11.1-3069.fce1f9718"
time="2022-06-23T13:56:25Z" level=info msg="Reconciling PravegaCluster store-ns/openshift-upgrade-pvg\n"
time="2022-06-23T13:56:25Z" level=info msg="updating statefulset (openshift-upgrade-pvg-pvg-seg) template image to 'asdrepo.isus.emc.com:8099/pravega:0.11.1-3069.fce1f9718'"
time="2022-06-23T13:56:25Z" level=info msg="Reconciling PravegaCluster store-ns/openshift-upgrade-pvg\n"
time="2022-06-23T13:56:25Z" level=info msg="statefulset (openshift-upgrade-pvg-pvg-seg) status: 4 updated, 4 ready, 4 target"
time="2022-06-23T13:56:25Z" level=info msg="segmentstore version sync has been completed"
time="2022-06-23T13:56:25Z" level=info msg="updating deployment (openshift-upgrade-pvg-pravega-controller) pod template image to 'asdrepo.isus.emc.com:8099/pravega:0.11.1-3069.fce1f9718'"
time="2022-06-23T13:56:25Z" level=info msg="failed to reconcile pravega cluster (openshift-upgrade-pvg): failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"openshift-upgrade-pvg\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1655992585.9127045,"logger":"controller.pravegacluster","msg":"Reconciler error","reconciler group":"pravega.pravega.io","reconciler kind":"PravegaCluster","name":"openshift-upgrade-pvg","namespace":"store-ns","error":"failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"openshift-upgrade-pvg\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
time="2022-06-23T13:56:25Z" level=info msg="Reconciling PravegaCluster store-ns/openshift-upgrade-pvg\n"
time="2022-06-23T13:56:26Z" level=info msg="statefulset (openshift-upgrade-pvg-pvg-seg) status: 4 updated, 4 ready, 4 target"
time="2022-06-23T13:56:26Z" level=info msg="segmentstore version sync has been completed"
time="2022-06-23T13:56:26Z" level=info msg="deployment (openshift-upgrade-pvg-pravega-controller) status: 2 updated, 2 ready, 2 target"
time="2022-06-23T13:56:26Z" level=info msg="controller version sync has been completed"

Importance

must-have

Location

upgrade.go

Suggestions for an improvement

Handle upgrade in this scenario as well