recover nil GRPCConnectionState by update failure #2561

akihikokuroda · 2022-01-10T02:42:54Z

Signed-off-by: akihikokuroda akihikokuroda2020@gmail.com

Description of the change:
This PR fixes the e2e test failures with the time out of the following message

waiting for catalog pod mock-ocs-main-j48j7 to be available (for sync) - NO_CONNECTION

When the grpc state change is processed first time, the Status.GRPCConnectionState of the CatalogSource instance is initialized in the syncConnection called by syncCatalogSources. When the CalalogSource instance update at the end of the syncCatalogSources is failed (because it has been modified), the Status.GRPCConnectionState is never initialized again. This change initializes the Status.GRPCConnectionState with the current values when is not initialized.

### CatalogSource sync entry
2022-01-05T19:11:07.689125083Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="syncing catsrc" id=aOM/E source=catalog-r84z2
2022-01-05T19:11:07.689219283Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="checking catsrc configmap state" id=aOM/E source=catalog-r84z2
2022-01-05T19:11:07.689755987Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="check registry server healthy: true" id=aOM/E source=catalog-r84z2
2022-01-05T19:11:07.689930688Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="registry state good" id=aOM/E source=catalog-r84z2
### first time grpc state change reported
2022-01-05T19:11:07.722578295Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="Got source event: grpc.SourceState{Key:registry.CatalogKey{Name:\"catalog-r84z2\", Namespace:\"a-5kxzr\"}, State:1}"
2022-01-05T19:11:07.722608396Z stderr F time="2022-01-05T19:11:07Z" level=info msg="state.Key.Namespace=a-5kxzr state.Key.Name=catalog-r84z2 state.State=CONNECTING"
2022-01-05T19:11:07.724631709Z stderr F time="2022-01-05T19:11:07Z" level=info msg="syncing catalog source for annotation templates" catSrcName=catalog-r84z2 catSrcNamespace=a-5kxzr id=xAxXL
2022-01-05T19:11:07.724645509Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="this catalog source is not participating in template replacement" catSrcName=catalog-r84z2 catSrcNamespace=a-5kxzr id=xAxXL
2022-01-05T19:11:07.724649709Z stderr F time="2022-01-05T19:11:07Z" level=debug msg="RemoveStatusConditions - request to remove status conditions did not result in any changes, so updates were not made" catSrcName=catalog-r84z2 catSrcNamespace=a-5kxzr id=xAxXL
### CatalogSource update failure
2022-01-05T19:11:07.724653909Z stderr F time="2022-01-05T19:11:07Z" level=error msg="UpdateStatus - error while setting CatalogSource status" error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"catalog-r84z2\": the object has been modified; please apply your changes to the latest version and try again" id=aOM/E source=catalog-r84z2
2022-01-05T19:11:07.724676309Z stderr F E0105 19:11:07.724190       1 queueinformer_operator.go:290] sync {"update" "a-5kxzr/catalog-r84z2"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "catalog-r84z2": the object has been modified; please apply your changes to the latest version and try again

Motivation for the change:

Closes #2560

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /doc
Commit messages sensible and descriptive

openshift-ci · 2022-01-10T02:43:04Z

Hi @akihikokuroda. Thanks for your PR.

I'm waiting for a operator-framework member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2022-01-13T19:38:18Z

/ok-to-test
/lgtm

awgreene · 2022-01-14T20:48:59Z

/approve

openshift-ci · 2022-01-14T20:49:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akihikokuroda, awgreene

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [awgreene]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-01-14T23:19:50Z

/retest-required