Flake: waiting for catalog pod to be available (for sync) - IDLE

[timflannagan]

This is flake is occasionally consistent throughout workflow runs, but difficult to reproduce locally. Link to a GHA workflow run that contains this: https://github.com/operator-framework/operator-lifecycle-manager/pull/2367/checks?check_run_id=3760863789.

There are a couple of test instances where this issue has propagated:

• User defined service account/with permission
• User defined service account/with no permission
• Operator Group/ static provider
• Install Plan/with CRD schema change/existing version is present in new CRD (deprecated field)

Creating catalog source scoped-catsrc-785ll in namespace scoped-ns-xrfbn...
Catalog source scoped-catsrc-785ll created
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
waiting for catalog pod scoped-catsrc-785ll to be available (for sync) - IDLE
...
• Failure [300.404 seconds]
User defined service account
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/user_defined_sa_test.go:25
  with permission [It]
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/user_defined_sa_test.go:82

  
  	Error Trace:	user_defined_sa_test.go:110
  	            				runner.go:113
  	            				runner.go:64
  	            				it_node.go:26
  	            				spec.go:215
  	            				spec.go:138
  	            				spec_runner.go:200
  	            				spec_runner.go:170
  	            				spec_runner.go:66
  	            				suite.go:79
  	            				ginkgo_dsl.go:238
  	            				ginkgo_dsl.go:221
  	            				e2e_test.go:57
  	Error:      	Received unexpected error:
  	            	timed out waiting for the condition
  	Test:       	User defined service account with permission

[timflannagan]

cc @dinhxuanvu