Tanzu Addon Controller backoff exponentially and blocks cluster creation for over 10min

[HanFa]

Bug description

When provisioning a management cluster, the Tanzu Addon Controller tries to reconcile the cluster CR. However, the ClusterBootstrapController keeps hitting the no matches for kind \"Package\" in version \"data.packaging.carvel.dev/v1alpha1\", which is expected before the kapp-controller has installed the Package CRD.

The recurring errors of ClusterBootstrapController causes an exponential delay (more than 10 minutes) in requeuing the next Cluster reconciliation. In the attached log, it is from 21:24:18.810764 to 21:35:42.178200.

Do we have a known approach to mitigate this issue?

Affected product area (please put an X in all that apply)

• ( ) APIs
• (x) Addons
• ( ) CLI
• ( ) Docs
• ( ) IAM
• ( ) Installation
• ( ) Plugin
• ( ) Security
• ( ) Test and Release
• ( ) User Experience
• ( ) Developer Experience

Expected behavior

The ClusterBootstrapController maybe should do a linear retry.

Steps to reproduce the bug

Management cluster creation path, according to #3600

Version (include the SHA if the version is not obvious)

Environment where the bug was observed (cloud, OS, etc)

Relevant Debug Output (Logs, manifests, etc)
addon-controller.log