[Alerting] alert types can't schedule actions within the same action group without being throttled

[pmuellr]

The way in which we have implemented throttling in the framework is insufficient for certain use cases.

The limitation

When an Alert Type schedules actions within a certain action group that scheduling will adhere to the throttling configured on the alert.
This is by design and allows a user to limit how often they're notified about the same unique identifier ("Alert"/"Alert instance") while their active action group remains unchanged.

The problem with this is that there are use cases (most recently in Maps) where an Alert Type wishes to assign the same unique identifier to the same action group, but wish to express that the detected condition has changed.

For example, imagine an Alert Type which tracks the movement of an entity through a map. When the detection algorithm finds that an entity is contained within a certain geo-fenced area, it schedules action in the "Contained" action group, using that entity's unique identifier. On the next execution the algorithm find that the entity has left the contained area, but entered a new contained area, scheduling actions in the "Contained" action group a second time.
This time this scheduling might get throttled bu the configured throttle, but the Alert Type would actually like to break throttling as it considers this a new condition.

This issue should provide a way for the Alert Type to "break through" the throttling.

Possible Approach

During discussion of the use cases described in issue #78981, one thought that came up was allowing an alert to explicitly break throttle. It kind of goes against the grain of the user being in control of throttling / muting, but seems like it might be useful in general as well. Gives the alert type a little more control, if they want it.

The idea is that the maps alert wants to be able to track entities moving in geofenced shapes, where the shapes are adjacent. You might imagine the alert has an action group for "contained in a shape", but then how do you let users know if they are still contained in a shape, but the shape has actually changed. We don't have a great, simple, obvious way to do that right now.

But if we did allow an alert to "break throttle", we can get this use case to work (we think). The alert itself would realize the contained shape changed, and arrange to break the throttle, which would allow the action groups to run, which would send notifications to the user that would presumably contain the new shape info.

I'm thinking this would be a new method in alerts/server/alert_instance/alert_instance.ts, rather than try to add it as an additional parameter on the scheduleActions() method, but not clear to me yet.

Then something would change here, where we filter out the currently throttled alert instances before actually scheduling the actions.

kibana/x-pack/plugins/alerts/server/task_runner/task_runner.ts

Lines 242 to 251 in 2cd2528

	await Promise.all(
	Object.entries(instancesWithScheduledActions)
	.filter(
	([alertInstanceName, alertInstance]: [string, AlertInstance]) =>
	!alertInstance.isThrottled(throttle) && !mutedInstanceIdsSet.has(alertInstanceName)
	)
	.map(([id, alertInstance]: [string, AlertInstance]) =>
	this.executeAlertInstance(id, alertInstance, executionHandler)
	)
	);

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)