[ReponseOps] error when two recovered actions run: Alert instance execution has already been scheduled

[pmuellr]

Created a rule with 4 actions - two for active, two for recovered. Rule type doesn't matter. Force the rule to go active, then allow to recover. When the recovered action processing is occurring the following error message will be logged:

[ERROR][plugins.alerting.index-threshold] Executing Rule default:.index-threshold:6a3d8c40-7bbc-11ed-be1a-31b626d2ea09 has resulted in Error: 
  Alert instance execution has already been scheduled, cannot schedule twice - 
  Error: Alert instance execution has already been scheduled, cannot schedule twice
    at Alert.ensureHasNoScheduledActions (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/alert/alert.ts:163:13)
    at Alert.scheduleActions (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/alert/alert.ts:146:10)
    at ExecutionHandler.run (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/task_runner/execution_handler.ts:228:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at /Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/task_runner/task_runner.ts:473:9
    at TaskRunnerTimer.runWithTimer (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/task_runner/task_runner_timer.ts:49:20)
    at TaskRunner.runRule (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/task_runner/task_runner.ts:462:5)
    at TaskRunner.run (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/alerting/server/task_runner/task_runner.ts:670:31)
    at TaskManagerRunner.run (/Users/pmuellr/Projects/elastic/kibana-prs/x-pack/plugins/task_manager/server/task_running/task_runner.ts:304:22)

Here's the rule I happened to use, which depends on #144689, and I used es-apm-sys-sim 1 4 -k to force the cpu value up and down. https://github.com/pmuellr/es-apm-sys-sim

esq-rule-schedule-twice.ndjson.zip

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

I believe this is due to the recent refactor that iterates over actions first, then alerts. With multiple recovery actions, this line:

kibana/x-pack/plugins/alerting/server/task_runner/execution_handler.ts

Line 228 in 7cee9ff

alert.scheduleActions(action.group as ActionGroupIds);

gets called multiple times for recovered alerts and the second time will throw an error.

[mikecote]

Moving to current iteration, we should get this backported to 8.6 if that release is affected as well. Thanks @pmuellr for the find!!

[ersin-erdal]

Tried locally, removing that line fixes the bug.