Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing notifications when using the Teams sink #1522

Open
crileroro opened this issue Aug 8, 2024 · 1 comment
Open

Missing notifications when using the Teams sink #1522

crileroro opened this issue Aug 8, 2024 · 1 comment

Comments

@crileroro
Copy link

Describe the bug
I am using label filtering in a Teams sink. I've noticed that if I create a list of labels in the include attribute, where the labels name are the same, just the first label value is taken and the second is ignored.

To Reproduce
On Prometheus, I have a metric containing a label workflow_name. Let's call that one workflow_count{workflow_name}. This is used on an alert called workflowFailure. Having these elements, I have defined the following customPlaybook:

 - name: "custom_playbook"
    triggers:
      - on_prometheus_alert:
          alert_name: workflowFailure
    actions:
      - create_finding:
          title: 'test $label.alertname'
          aggregation_key: workflowFailure
          severity: MEDIUM
          description: |
            This is a test of severity Medium

This is the scope section I am using in the Teams sink:

sinksConfig:
  - ms_teams_sink:
      name: ms_teams
      webhook_url: my_url
      scope:
        include:
          - labels:
              - "workflow_name=monitoring-wf-1"
              - "workflow_name=monitoring-wf-2"

If my understanding is correct, the sink will be used when the workflow_name label is either monitoring-wf-1 OR monitoring-wf-2. If the cluster fires 2 alerts, one for monitoring-wf-1 and one for monitoring-wf-2, I would expect to get two different notifications, one for each workflow. However, I only get notified for monitoring-wf-1. If I switch the order of the labels as follows, I get notified for monitoring-wf-2 only instead. My feeling is that if the label key is duplicated (e.g, workflow_name), Robusta will take the first one it finds:

      scope:
        include:
          - labels:
              - "workflow_name=monitoring-wf-2"
              - "workflow_name=monitoring-wf-1"

Expected behavior
I should receive two different notifications.

@Avi-Robusta
Copy link
Contributor

Avi-Robusta commented Oct 13, 2024

Hi @crileroro
This wont solve your issue but can you try:

scope:
        include:
          - labels:
              - "workflow_name=monitoring-wf-2"
          - labels:
              - "workflow_name=monitoring-wf-1"

From the docs it looks like your configuration might be checking for label workflow_name=monitoring-wf-2 AND workflow_name=monitoring-wf-1 so in the past you only would have received alerts if both labels exist in your config
https://docs.robusta.dev/master/notification-routing/configuring-sinks.html#routing-alerts-to-specific-sinks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants