[Alerting] getAlertStatus() should exhaustively search through event log

[pmuellr]

see: #68437 (comment)

Currently the alert client's getAlertStatus() method (added in PR ^^^) only gets up to 10K events to process to determine the status. That may not be enough, for alerts with many active instances - it may drop some old data.

We should probably change this to exhaustively search the event log instead - although perhaps some reasonable limit on the start date should be added. We'd have to change the way the processing of the event log works.

• change the event log query to sort ascending, and process the first "page" of events coming in
• after processing that "page" of events, get the next page and continue

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[YulNaumenko]

@pmuellr could this issue to be related to the PR I've closed and the closed issue?

[pmuellr]

@pmuellr could this issue to be related to the PR I've closed and the closed issue?

Good question, and I think no. We were still using a time-range to search through the event log, so there's always a possibility we didn't go quite far enough back.

This issue I think points to a better approach - issue #93704

The idea is that we'd store the "new-instance" date in the instance state, and arrange to write it out with probably the active-instance and recovered-instance events, so we NEVER have to search for the new-instance events. This would allow us to get the date ranges even if the older events were deleted via ILM.