[RAC] Alerts as Data Bulk Insert

[spong]

This issue is for discussing the architecture/implementation for writing out Alerts as Data.

Relevant implementations from within detections include:

• bulk_create_threshold_signals.ts
• bulk_create_ml_signals.ts
• search_after_bulk_create.ts
• single_bulk_create.ts

And a plethora of other utilities and helpers found in:
kibana/x-pack/plugins/security_solution/server/lib/detection_engine/signals/

The above implementations can be spread out as they cover many complex use cases, like creating alerts for aggregations, EQL sequences, alerts on alerts, and logic for preventing the creation duplicate alerts (among others :). Some patterns have started forming as we've been adding new rule types and features over the last year, but until now we've yet to have the opportunity to abstract further and clean up the control flow.

As discussed with @mikecote, @sqren and @tsg, a library implementation providing a hook for alert creation that each rule could provide their own implementation to (and call into other library utils like exceptions, deduplication, etc) would go a long way in extending the initial implementation within Detections.

Open questions

• How will the existing paradigm of alert/alert instances port over to alerts as data? For example, in the o11y use case, a CPU Threshold Exceeded rule may write an alert, then later update that alert if it is triggered again. Will we follow this same paradigm, or are alerts immutable except for assignment/status like in the security use case? If the latter, how will what is now many alerts for one incident be displayed in the triage workflow since we don't support groupings/aggregations? Each new trigger creates a new alert with aggregate information from the predecessors and automatically closes them so there's only ever one active alert? Building block alerts for each trigger, and the shell alert is mutable?

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[pmuellr]

This may not be relevant, but thought I'd mention it.

For the current event log, we currently queue up events to write, in bulk, async. However the public interface is a simple synchronous logEvent(event) style call that returns nothing. We buffer over time and space (1 sec or 100 events, whichever happens first). We also handle trying to get these written out during an "normal" Kibana shutdown (plugin shutdown is await'd by the platform (with a timeout)!) - otherwise there's a good chance we'd lose the last few records that were left in the buffer. As of 7.11 or so, you should start seeing BOTH the event log startup event AND the event log shutdown event written to the event log - we've always been writing the event log shutdown event, but didn't handle the actual "write the last buffer during shudown" till then, so before that, it was never written.

We've always treated the event log data as "not critical" - for the cases we query it in alerting (to show alert details), we handle cases where data is missing for some reason. Beyond bugs / timing issues / etc, that reason could be that the data got ILM'd away in a delete phase. I'm guessing we want to make the "alerts as data" a little more "critical" :-), but not quite sure what that means, because if we buffer the data, and then Kibana crashes with an OOM or SIGSEGV, you're going to lose that last set of buffered data.

All that code (there's not much) to deal with the event log buffering is here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/event_log/server/es/cluster_client_adapter.ts#L52-L106

[peluja1012]

Closing in favor of Alerts as Data RFC doc.