[Security Solution][Alerts] Use dynamic mappings to create smaller templates & mappings

[tsg]

The Fleet and Elasticsearch teams have optimized the data ingestion mappings to use dynamic mappings as much as possible: elastic/elasticsearch#64978 Here is the current version of the template: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/data-streams-mappings.json

We should consider doing the same for alerts as data.

The idea is to take advantage of the following observations:

• most ECS fields are keywords, so if we default to the keyword type, the template size can be dramatically reduced
• ECS has conventions for IP fields (*.ip) and text fields (*.message)
• the rest of the fields can show up explicitly in the mapping.

In addition to the benefit of having smaller templates, this means that all the fields that don't show up explicitly (keywords) are zero-cost when they are not used. This reduces the cost of assuming the whole of ECS is the base of the Alert schema.

We need to be careful, though, to not cause mapping conflicts, because we're copying data from disparate indices that can be in conflict with one another. I think the following approach would be feasible:

• The Alert mappings use dynamic fields for keywords and other special cases
• The alert.original_event field is set to be not indexed at all or flattened
• The Kibana code that creates the Alert documents puts all the ECS + well known alerting fields at the top level, and any other field under alert.original_event.
• When copying the ECS + well-known fields, the code must verify that the fields have the correct types (string vs object vs number) and ignore the fields that don't match. This ensures that the resulting mapping is consistent with ECS.
• Every other field goes under alert.original_event where it can't cause conflicts.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jportner]

@tsg did you mean to use the Team:SIEM label for this?

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[tsg]

Argh, yes, sorry.

[tsg]

FYI on another evolution on the Integration side: elastic/package-spec#178 The approach above doesn't have the same problem for us, because the top-level fields are always well-known.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

Hey everyone, FYI ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.

[marshallmain]

This issue primarily affects security solution alerts indices since those indices are currently the only rule registry-based indices that use the full ECS mappings.