[Alerting] Provide support for derivative threshold alerts

[jeffvestal]

Describe the feature:
I would like to create alerts that trigger when a value changes a certain percent over X minutes compared to the previous value

Describe a specific use case for the feature:
In operations it can be very useful to know when certain metrics start changing even before reaching critical ceiling thresholds. Being able to identify when things are changing in your environment as early as possible. This is frequently accomplished with derivative / delta / rate of change calculations.

• The CPU utilization for any host jumps 25% over a 3 minute period - process may be starting to hang
• Free disk space dropped 30% over the last 5 minutes - logging rate has increased
• The number of failed login attempts to the portal has increased 30% over 2 minutes - users are starting to be unable to login in to the platform

While anomaly detection is often a great choice to identify what is usual, being able to set certain known threshold to trigger an alert on is often needed / requested.

Watcher supports this type of alert through the use of pipeline aggregations

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[ghudgins]

while probably not the primary way, one way we could solve this is the lens integration with alerting #71150

[gmmorris]

Removing Theme: rac as this will not be delivered as part of RAC and is rather a general Alerting ER.
I just want to make sure this doesn't get lost in the RAC backlog.

[Cclleemm]

Hello,

do you have any news about the integration of this feature?
Like you @jeffvestal I would like it to be native in "metrics threshold alert".

Otherwise, the other possibility could be to use a query (like below) and get the derivation aggregations. But unfortunately it is not possible (as far as I know) to do a Custom Lucene Query Alert with an aggregation.

Calculation of derivative thread increase threshold for a specific service

GET .ds-metrics-apm*/_search
{
  "size": 0,
  "query": { 
     
    "bool": { 
      "must": [
        { "match": { "service.name":   "xxx" }}
      ], 
       "filter": [ 
        { "range": { "@timestamp": { "gte": "now-5m/m" }}}
      ]
    }
  },
  "aggs": {
    "sales_per_month": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "minute"
      },
      "aggs": {
        "thread_count_average": {
          "avg": {
            "field": "jvm.thread.count"
          }
        },
        "thread_count_derivation": {
          "derivative": {
            "buckets_path": "thread_count_average" 
          }
        },
        "sales_bucket_filter": {
          "bucket_selector": {
            "buckets_path": {
              "threadCountDerivation": "thread_count_derivation"
            },
            "script": "(params.threadCountDerivation ?: 0) > 10"
          }
        }
      }
    }
  }
}

Do you have an alternative at the moment, other than the "machine learning" features?

Thank you