PTEUDO-1693 automation of CRD disabling flux and helm pruning

[drewwells]

Adds annotation and labels to prevent helm and/or flux from deleting CRDs when a helm upgrade indicates that the CRDs should delete them.

DEMO

helm upgrade --install --create-namespace --namespace db-controller db-controller-crds helm/db-controller-crds --debug --wait
history.go:56: [debug] getting history for release db-controller-crds
...
client.go:486: [debug] Starting delete for "db-controller-crds-pre" Pod
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Pod db-controller-crds-pre with timeout of 5m0s
client.go:740: [debug] Add/Modify event for db-controller-crds-pre: ADDED
client.go:799: [debug] Pod db-controller-crds-pre pending
client.go:740: [debug] Add/Modify event for db-controller-crds-pre: MODIFIED
client.go:799: [debug] Pod db-controller-crds-pre pending
client.go:740: [debug] Add/Modify event for db-controller-crds-pre: MODIFIED
client.go:799: [debug] Pod db-controller-crds-pre pending
client.go:740: [debug] Add/Modify event for db-controller-crds-pre: MODIFIED
client.go:799: [debug] Pod db-controller-crds-pre pending
client.go:740: [debug] Add/Modify event for db-controller-crds-pre: MODIFIED
client.go:794: [debug] Pod db-controller-crds-pre succeeded
client.go:393: [debug] checking 0 resources for changes
client.go:442: [debug] Deleting CustomResourceDefinition "databaseclaims.persistance.atlas.infoblox.com" in namespace ...
client.go:453: [debug] Skipping delete of "databaseclaims.persistance.atlas.infoblox.com" due to annotation [helm.sh/resource-policy=keep]
client.go:442: [debug] Deleting CustomResourceDefinition "dbroleclaims.persistance.atlas.infoblox.com" in namespace ...
client.go:453: [debug] Skipping delete of "dbroleclaims.persistance.atlas.infoblox.com" due to annotation [helm.sh/resource-policy=keep]
upgrade.go:428: [debug] waiting for release db-controller-crds resources (created: 0 updated: 0  deleted: 0)
wait.go:48: [debug] beginning wait for 0 resources with timeout of 5m0s
upgrade.go:171: [debug] updating status for upgraded release for db-controller-crds
Release "db-controller-crds" has been upgraded. Happy Helming!

kubectl api-resources --api-group=persistance.atlas.infoblox.com
NAME             SHORTNAMES   APIVERSION                          NAMESPACED   KIND
databaseclaims   dbc          persistance.atlas.infoblox.com/v1   true         DatabaseClaim
dbroleclaims                  persistance.atlas.infoblox.com/v1   true         DbRoleClaim

[drewwells]

DEMO on dev-hub
https://github.com/Infoblox-CTO/deployment-configurations/pull/65708

[kd7lxl]

LGTM

[daniel-garcia]

LGTM

[jcookson-infoblox]

lgtm