Make cluster resource exclusion configurable

[2opremio]

Describe the feature

We already exclude metrics resources (see #2606 , however, that is hardcoded, and (as seen in #2633 ) it may be useful to exclude additional resources.

What would the new user story look like?

Users are provided with a new fluxd flag, say --k8s-exclude-resource=apigroup/version/kind.

Users can supply:

• Fixed values like --k8s-exclude-resource=coordination.k8s.io/v1/Lease
• Multiple values like --k8s-exclude-resource=coordination.k8s.io/v1beta1/Lease,coordination.k8s.io/v1/Lease
• Glob experssions like --k8s-exclude-resource=coordination.k8s.io/*/Lease , --k8s-exclude-resource=authrequests.dex.coreos.com/*

Expected behavior

Flux won't attempt to obtain resources of that kind from the cluster.

[2opremio]

I am now thinking that this could probably achieved by simply disallowing Flux to list/get resources through RBAC rules.

[primeroz]

The RBAC way make sense, I will give it a try as soon as i can but as long as it does not error out in the logs that is as good as a filter in the args to me.

Might be a bit harder to handle in terms of new users and documentation though ?

It would be easier to document something

• RBAC allows pulling everything, filter args obviously remove stuff from flux downloading ( and applying ? ) resources
  rather than
• This is a base RBAC, extend if you need more resources to be handled by flux
  assuming that the default example rbac would be somewhat limited and operators would have to extend it on their own .. but as long as we get 403 in the logs and errors in the metrics it should be doable.

[ellieayla]

Nice; some resources that Flux isn’t expected to be manipulating (eg Pods and ReplicaSets) can be excluded for the common-case. That might make for a good example in the docs.