[RAC][Discuss] Support for user-defined field mappings in .alerts indices

[marshallmain]

Advanced users of the Security Solution have a history of wanting additional custom fields mapped in the .siem-signals indices. Even though the Security Solution copies the entire source document into the generated alert, fields that were mapped in the original document may not be mapped in the alert document, preventing users from queries on .siem-signals from using those fields effectively.

Users have sometimes seen this limitation and decided to augment the .siem-signals index template with their own custom fields. However, this causes problems for them when they upgrade Kibana as we often add additional fields to the template - when we add more fields, we overwrite the existing template and in the process wipe out any changes that users made to the template. The guidance we typically give to users in this situation right now is that (1) what they're doing is not supported, so it's inherently risky, and (2) if they still want to continue augmenting fields in a risky way, they may have better success by creating a separate template and using the order parameter (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html#multiple-templates-v1) to apply their custom fields separately from the built-in mappings shipped by the Security Solution. However, the order parameter is going away with index templates v2 and essentially replaced by component templates.

To avoid situations where users modify the .alerts index templates directly and are surprised when Kibana upgrades clobber their customizations, it would be nice to decide on and document a way for advanced users to augment the mappings. 2 solutions have been proposed so far:

1. Guide users towards analyst defined runtime fields on .alerts indices. Kibana will add the runtime mappings at query time so most(?) searches and visualizations on those user defined fields should work. This is probably the more user-friendly approach, but provides less flexibility for advanced users since the fields must be runtime and would not be part of the actual index mapping.
2. Provide a mechanism for users to define their own component template that becomes part of a .alerts index template. This could be accomplished by defining a standard component template name (e.g. (4) in [RAC][Meta] Consolidate the two indexing implementations in rule_registry plugin #101016 (comment)). This approach is less user-friendly and requires more knowledge of Elasticsearch, but would provide the most flexibility to advanced users.

cc @oatkiller @tsg @MikePaquette @dgieselaar @jasonrhodes @weltenwort

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[jen-huang]

For awareness, proposal 2 is similar to what we implemented in in Fleet recently, which installs a @custom component template for every index template we install: #101769

The initial UX is a bit clunky as users will need to go into Stack Management and apply their custom settings/mappings/ILM policies, etc. But the idea is that in the future we can iterate towards a custom UI that is "built in" into Fleet/Integrations that modifies these custom component templates in the background.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[twanva]

Hi, we are also interested in this issue. It's very frustrating to work with the Alerts overview from the Security Solution and filter out fields, which then result in nothing happening or all alerts being filtered out because the field "does not exist".

[oatkiller]

@marshallmain Is this issue still valid?

[threatangler-jp]

We have had success using run-time mappings to work around this issue. Thank you to @marshallmain for the tip in the slack community.

We should clarify something though. The description above suggests this only impacts custom mappings. Since Elastic has not created all standard ECS mappings in the .alerts index, this leaves it to the customer to do so. Some customers will truly have custom mappings, but in our situation, we are just trying to apply the standard ECS mappings because they are not there natively.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[rlapond]

hi,

We'd also like to make use to custom field in the .siem-signals indices. Our Analists mainly want to use it for dashboarding/ filters in searches.