Installplan failed because of the error updating CRD

[Daniel-Fan]

Bug Report

What did you do?
Install a new operator from OperatorHub or upgrade the existing installed operator.

What did you expect to see?
the operator is installed or upgraded successfully

What did you see instead? Under which circumstances?
It is an intermittent issue, did not show up every time.
The operator is showing pending status.
The installplan is failed and reports the following status

conditions:
    - lastTransitionTime: '2021-03-10T16:12:09Z'
      lastUpdateTime: '2021-03-10T16:12:09Z'
      message: >-
        error updating CRD: commonservices.operator.ibm.com: Operation cannot be
        fulfilled on customresourcedefinitions.apiextensions.k8s.io
        "commonservices.operator.ibm.com": the object has been modified; please
        apply your changes to the latest version and try again
      reason: InstallComponentFailed
      status: 'False'
      type: Installed
  phase: Failed

Environment

• operator-lifecycle-manager version: OCP 4.6.16

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a7d2cd3ae6e1d2e0bab3682f86f916ca93d93aaedb04cff478a72063557f090

• Kubernetes version information:

• Kubernetes cluster kind:

Possible Solution
I think when the olm-operator updates the CRD of operator, it failed to match the resourceversion.

The User manual workaround is to uninstall the pending operator and re-install it from the OperatorHub

[horis233]

Considering the installplan can't recovery by itself, users have to delete it and reinstall it.

Could OLM add some re-try logic for installplan instead of just return the error and block the installation?

Or

For this specific problem, could we use the patch command instead of the update when updating existing resources?

https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/operators/catalog/step.go#L159

[benluddy]

Thanks for taking the time to file this issue. Retries for InstallPlan step errors are coming soon (tracked downstream by https://issues.redhat.com/browse/OLM-2114), which should cover 409s as well as transient I/O errors and API server unavailability. In the meantime, you should also be able to patch .status.phase to "Installing" in order to make the catalog operator continue the existing plan.

[horis233]

@benluddy

Thanks for your quick response. Patching .status.phase back to "Installing" sounds like a good solution.

[exdx]

Addressed in #2090 which added retries to installplan execution. That should help the install succeed in spite of transient errors