[alerting] http status 422 from webhook action to opsgenie

[pmuellr]

Trying to use OpsGenie from the webhook connector, and am seeing http 422 responses from my simple test.

Here's some of the data from the alert/action:

    {
        "id": "e00f4304-1e87-4161-9bf8-c8413825299f",
        "actionTypeId": ".webhook",
        "name": "opsgenie",
        "config": {
            "method": "post",
            "url": "https://api.opsgenie.com/v2/alerts",
            "headers": {
                "Authorization": "GenieKey <api key here>,
                "Content-Type": "application/json"
            }
        },
        "isPreconfigured": false,
        "referencedByCount": 1
    },

            "actions": [
                {
                    "actionTypeId": ".server-log",
                    "params": {
                        "level": "info",
                        "message": "{{context.message}}"
                    },
                    "group": "threshold met",
                    "id": "2bb6ab6b-f3b4-46c2-a581-7163afdb8749"
                },
                {
                    "actionTypeId": ".webhook",
                    "params": {
                        "body": "{\n    \"message\": \"{{{context.message}}}\"\n}"
                    },
                    "group": "threshold met",
                    "id": "e00f4304-1e87-4161-9bf8-c8413825299f"
                }
            ],

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

For reference, the API I tried is described here - https://docs.opsgenie.com/docs/alert-api#create-alert

I created a 14-day OpsGenie trial to try this out, not sure if we have an elastic account for OpsGenie or not - it's an Atlassian product.

[pmuellr]

Note that I originally opened this from an SDH where the customer thought the order of the required Authorization and Content-Type headers was significant - specifically that Authorization had to come first, otherwise a 422 response would be returned.

I was finally able to get OpsGenie to work from Kibana, the only change from the info ^^^ is that I had to use an Integration API key, not just a plain old API key (they support multiple types of API keys). To get to the point where you can create an Integration API key, you need to add a Team to your account, then you'll be able to create an Integration API key for the team.

Using that key, I was able to post to Opsgenie from an alert, with the headers in both orders.

I suspect what the customer was seeing were related to some of the other issues we've had with webhook headers:

• Fixed Webhook connector doesn't retain added HTTP header settings #71924
• [Alerting] Updating Webhook with removed headers doesn't work as expected #71995

It also seems to be the case that OpsGenie returns 422 in a lot of cases - it's basically a specialized 400 response you'd see from most other services - I think they separate "syntax" issues into 400's, and "semantic" issues into 400.

[pmuellr]

going to close this for now, perhaps some breadcrumbs for future issues with OpsGenie ...

[pmuellr]

Thought I'd clarify one bit here - I wasn't able to actually reproduce the customer's error of just changing the headers and then seeing a 422 response code. However, it seems to me that the referenced issues with headers for the webhook action is most likely at fault here.

I did notice that 422 seemed to be a fairly "common" response code returned from OpsGenie - the case I saw was when I used an inappropriate API key (wasn't an Integration API key, just a plain API key), so it is possible that there was some other issue involved in the customer's environment. I don't remember now if the response body contained more details about the error; if so, in the future, we will be exposing more of this sort of error information in the UI, in future releases. There are a couple of ways of getting this info today using some of our HTTP APIs, if required - LMK for more info on this.