Large shard size alert failing because of missing definition for aggregation.

[nicenemo]

Kibana version: 7.12.1

Elasticsearch version: 7.12.1

Server OS version: Debian 10

Browser version: 90.0.4430.93 (Official Build) (64-bit)

Browser OS version: Windows 10 Pro

Original install method (e.g. download page, yum, from source, etc.): apt

Describe the bug:
I get an alert on the Large shard size alert definition not being OK.

An error occurred when running the alert.
[parsing_exception] Missing definition for aggregation [over_threshold], with { line=1 & col=561 }

Steps to reproduce:
I have no idea in what version this started or whether I caused it. If I did something stupid please tell me.

Expected behavior:

• Not seeing this alert on the alert but a working alert.
• Having a query or script to fix it would be nice too.

Screenshots (if relevant):

Errors in browser console (if relevant):
not relevant
Provide logs and/or server output (if relevant):

Fltered Kibana logs

The alert is checked every minute;I get a lot of these.

...
n for aggregation [over_threshold], with { line=1 & col=561 }"}
{"type":"log","@timestamp":"2021-04-28T11:31:22+00:00","tags":["error","plugins","alerts","plugins","alerting"],"pid":10312,"message":"Executing Alert \"a8d20300-a814-11eb-9025-99701d4cf078\" has resulted in Error: [parsing_exception] Missing definition for aggregation [over_threshold], with { line=1 & col=561 }"}
{"type":"log","@timestamp":"2021-04-28T11:31:25+00:00","tags":["error","plugins","alerts","plugins","alerting"],"pid":10312,"message":"Executing Alert \"5b4f3de0-9ed3-11eb-bd13-f7f860a04220\" has resulted in Error: [parsing_exception] Missing definition for aggregation [over_threshold], with { line=1 & col=561 }"}
{"type":"log","@timestamp":"2021-04-28T11:32:25+00:00","tags":["error","plugins","alerts","plugins","alerting"],"pid":10312,"message":"Executing Alert \"a8d20300-a814-11eb-9025-99701d4cf078\" has resulted in Error: [parsing_exception] Missing definition for aggregation [over_threshold], with { line=1 & col=561 }"}
}
...

Any additional context:

3 node cluster. with Logstash on one and Kibana on another.

[crowels]

I am seeing the same issue with upgrading from 7.11.2 to 7.12.1. Shows new alert available in pop-up bottom right of page and on alerts page displays same error for Shard Size alert.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[ymao1]

@elastic/stack-monitoring Is this issue fixed with #99159?

[igoristic]

@ymao1 Yep! Thank you for referencing the issue. I will close this ticket, but feel free to reopen it