Slack workflow webhook always results in "failed to send Slack message: unexpected empty response"

[kylehounslow]

What is happening?

Using Slack workflow webhook results in errors thrown in Grafana server code, even though Slack messages are sent successfully.
Example logs:

logger=ngalert.notifier.alertmanager org=1 t=2024-04-24T18:43:15.211109257Z level=error component=alertmanager orgID=1 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="slack/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: failed to send Slack message: unexpected empty response"

Slack message sent successfully:

Important note:

If running Grafana in HA mode, this results in alerts being sent multiple times, since HA peers think that their peers failed to send the notification, even though it was sent successfully.

What is expected to happen?

• Webhook "200 ok" responses are gracefully handled without error.
• Using Slack webhook contact point when running in HA mode results in only 1 notification sent.

Steps to reproduce

1. Create a Slack webhook (docs)
2. Run a grafana server locally (e.g. docker run --rm -it -p "3000:3000" grafana/grafana-oss:10.4.1)
3. Create a Slack contact point and enter Slack webhook URL created in step 1
4. Set default notification policy to use Slack contact point
5. Create a new Alert with a threshold that causes it to always be firing (e.g. >-1)
6. Observe errors in Grafana server logs

Other Notes

I believe the problematic code is here, where a JSON object is always expected in the response. However from the Slack docs, it states that plaintext "ok" will be returned on success: https://api.slack.com/messaging/webhooks#handling_errors

in most cases you'll receive a "HTTP 200" response with a plain text ok indicating that your message posted successfully

[KaplanSarah]

Can we get a copy of the slack config? Is there a proxy or other network service in between grafana and slack? We have failed to reproduce this with the steps above.

[kylehounslow]

@KaplanSarah thanks for the response. See screenshot below of Slack config. All collapsed items are default/blank.

Corresponding YAML export:

apiVersion: 1
contactPoints:
    - orgId: 1
      name: slack-test
      receivers:
        - uid: cdl0ljfgyzzepd
          type: slack
          settings:
            url: https://hooks.slack.com/workflows/<REDACTED>
          disableResolveMessage: false

Is there a proxy or other network service in between grafana and slack?

No, I do not believe there is a proxy between myself and https://hooks.slack.com.

One more thing that might be important is this is a Slack "workflows" webhook. https://slack.com/help/articles/16583775096083-Automations--What-is-a-Slack-workflow

---

Let me know if you need any further info, thanks!

[yuri-tceretian]

@kylehounslow can you share more details about how you created the workflow? I am not familiar with this way of using Slack but if I have a way to reproduce it locally, I can fix the problem in the integration.