fix: Health check from lister not apiserver

[cwdsuzhou]

Fixes #11374

Motivation

Reduce the List check from controller. IMO, it is not reasonable to list from apiserver for health check. Informer is not so reliable and if informer is not healthy, the controller may exit or resync from apiserver.

Modifications

Change the health check logic from apiserver to the local cache

Verification

[terrytangyuan]

See discussions in #11374

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[terrytangyuan]

Bump

[cwdsuzhou]

Since we have disscussed #11374, can we work on this PR now?

@terrytangyuan

[terrytangyuan]

I am ok with this change. Let's watch for any issues once merged/released

[astraw99]

I am ok with this change. Let's watch for any issues once merged/released

Raised a PR #11770 to fix an issue #11769 introduced from this.
PTAL thanks.

[juliev0]

@cwdsuzhou When you tested this, were you able to successfully execute line 37 while catching a Workflow that hadn't been reconciled yet? We noticed a panic occurring on this line, when the *Unstructured type in the Informer gets type asserted to a *Workflow type instead. It seems like the Informer holds *Unstructured types, so I'm not sure if this type conversion ever works. If you have any thoughts, let me know. Thanks!

[agilgur5]

We noticed a panic occurring on this line, when the *Unstructured type in the Informer gets type asserted to a *Workflow type instead. It seems like the Informer holds *Unstructured types, so I'm not sure if this type conversion ever works.

For posterity, this was followed up in #12326 and #12353