[Alerts] Calling alertInstanceFactory and then scheduling no actions causes a recovery notification loop

[Zacqary]

Root cause of #86507 and #91035. When an alert executor calls alertInstanceFactory but then fires no actions, the alert seems to perpetually trigger a recovery "status change," and will continue to send notifications over and over.

This can be reproduced in 7.11.0 with either the Inventory or Metric Threshold alert types.

A workaround is to only call alertInstanceFactory behind a conditional, when you're sure that scheduledActions is going to be called. Unfortunately, we don't have an easy linting rule for that, so it would be safer to fix this at the alerting plugin level.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[Zacqary]

To reduce this to minimum complexity, test this by creating an alert executor that just does:

const alertInstance = alertInstanceFactory('example');
const shouldFire = // Run some code to read an ES document that returns either `true` or `false`
if (shouldFire) {
  alertInstance.scheduleActions(
      // ...
  );
}

Get the alert to fire once, then let it recover. It will continue to dispatch the "Recovered" action every time the executor runs.

[pmuellr]

Tried a repro with my typical setup using the index threshold - not able to repro. However, it only creates the instance when needed, so presumably doesn't have this problem:

kibana/x-pack/plugins/stack_alerts/server/alert_types/index_threshold/alert_type.ts

Lines 195 to 196 in e3f6729

	const alertInstance = options.services.alertInstanceFactory(instanceId);
	alertInstance.scheduleActions(ActionGroupId, actionContext);

[Zacqary]

Duplicate of #91117