Notifications - information content and filtering of info alerts

[tbomberg]

Hi,
I am currently working on the migration of our flux(v1)/helm-operator setups to flux(v2)/Toolkit.

regarding notifications i am struggling to get a reasonable configuration regarding of information and volume.

With flux(v1) we use the (--connect) functionality together with fluxcloud to post informational messages to a Teams channel, we created for each cluster.

With flux(v1) the message regarding the sync o a git commit contains

• a list of commits with commit messages from the repository
• a list of kubernetes objects updated
• error message in case something. went wrong

I found the information content of these alerts very valuable.

Now with flux2 it seems that I will be missing some of this information.

I do get error information from the controllers if something went wrong.
That is fine, maybe even more then before since Helm is covered as well.
The repeating alerts on retries are limiting the usability as already mentioned in this discussion #661 .

But for the info messages I am currently missing some valuable information

• source-controller: on a repo update I only get the information that an updated happened.
  Nothing about the change itself
• kustomize-controller:
  • message about the update of kubernetes objects, but if GC is enabled, all objects are always updated.
    I have seen in Slack notification is misleading #626 , that it has to be that way, but how did flux(v1) manage get get this information out even with GC enabled.
  • For a deleted object I get the information (I think this was not posted in flux(v1))
  • I also get alerts for every positive health check, but I have not found a way to filter certain alerts.

Did I overlook something? Is the information available but just hidden by the provider?
Can we get closer to the former message value?

One thing I would suggest is to add some kind of filter for the info events, either by name or
to classify into read/write operations.

Also currently the alerts show no information of the kind of toolkit object (or controller) they come from (e.g. GitRepository or Kustomization), just the name.namespace. This might be a nice add.

[hiddeco]

One thing I would suggest is to add some kind of filter for the info events, either by name or
to classify into read/write operations.

One thing that would help here is my suggestion in fluxcd/pkg#56 (comment)

[stefanprodan]

One thing that would help here is my suggestion in fluxcd/pkg#56 (comment)

I don't think so, @tbomberg want's to filter the info notifications not Kubernetes events.

Also currently the alerts show no information of the kind of toolkit object (or controller)

The controller-name shows up as the username, see https://toolkit.fluxcd.io/_files/slack-info-alert.png

I also get alerts for every positive health check, but I have not found a way to filter certain alerts.

This is a bug tracked in fluxcd/kustomize-controller#236

but how did flux(v1) manage get get this information out even with GC enabled.

Flux1 GC is very different to Flux2 GC. The v2 GC tracks objects without querying the Kubernetes API for each Kind, this means GC can be run under a restricted service account with no cluster wide access (unlike v1 that needs a cluster role binding).

[tbomberg]

The controller-name shows up as the username, see https://toolkit.fluxcd.io/_files/slack-info-alert.png

Unfortunately using the MS-Teams provider the username is always the name of the Incoming Webhook that is used for this, so I would need to create a separate webhook + provider for every controller, but that would be a workaround.

[stefanprodan]

There is a summary field that you can set in the Alert object, this can be anything: the controller name, cluster name, region, zone, some other id. Would this work for you?

[tbomberg]

There is a summary field that you can set in the Alert object, this can be anything: the controller name, cluster name, region, zone, some other id. Would this work for you?

Ok this adds this part of information, but honestly it does not look nice.

There I would rather go for adding webhooks.

But what about prefixing the resource name with the kind:

e.g. kustomization/prometheus-stack.flux-system

[stefanprodan]

But what about prefixing the resource name with the kind:
e.g. kustomization/prometheus-stack.flux-system

Please create an issue in notification-controller for this.