[Security Solution] data_stream.namespace field mapping missing from .internal.alerts-security.alerts-default indices

[ccdta]

Describe the bug:
the data_stream.namspace field cannot be filtered on .internal.alerts-security.alerts-default indices

Kibana/Elasticsearch Stack version:
8.7.0
Server OS version:

Browser and Browser OS versions:

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.):

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Steps to reproduce:

1. Generate Security Alerts
2. Kibana > Security > Alerts: if there are security alerts generated and a data_stream.namespace within the alert context. apply a KQL filter: data_stream.namespace: *
3. Also in Discover, the running this filter on the .internal.alerts-security.alerts-default documents will return no results

Current behavior:
The data_stream.namespace, dataset, type in the .internal.alerts-security.alerts-default indices are not searchable. They do appear in each security alert documents.

We added these fields as a runtime field in the native .alerts-security.alerts-mappings components template and reindex, which resolved the issue. However, after every cluster upgrade it resets and we have to repeat this process.
{
"runtime": {
"data_stream.namespace": {
"type": "keyword"
}
}

Expected behavior:
These fields should be searchable for the alerts index

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[MilkyEsquire]

@ccdta We are currently experiencing the same limitation with multiple clients and needing to filter the alerts. Can you possibly just expand on your current work around you are using? Where you are putting the fields etc? As this would be handy as a workaround for us too.

[ccdta]

@MilkyEsquire
Workaround:

1. In the 'alerts-security.alerts-mappings' component template, add the runtime field for data_stream.namespace (screenshot below)
2. Rollover your current .internal.alerts-security.alerts-default-00000* index
3. Create an index clone of an .internal.alerts-security.alerts-default-00000* index, however, now add the runtime field to the mapping
4. Reindex an .internal.alerts-security.alerts-default-00000* index to the new cloned index
5. The filters should now work

Note: Whenever there is a cluster upgrade. The component template will revert and the workaround has to be applied again.

[MilkyEsquire]

@ccdta
Great thanks for that, worked nicely. We submitted a ticket to Elastic Support around this missing feature a couple of months back also, so hopefully they will add this functionality in!

[ccdta]

Elastic team, any updates on this?

[yctercero]

@elastic/response-ops is this something you guys could look at?

[ymao1]

@yctercero We originally included the data_stream.* mappings in the ECS component template that the alerts indices reference but ran into issues because the detection rules use the mappings to determine which fields to copy from the source indices to the alert indices. Because the mapping for these are constant_keyword and the alert indices hold data from multiple different sources, we ran into issues because copying from multiple sources would break the constant_keyword mapping. We then omitted any fields mapped as constant_keyword (which ends up just being the data_stream.* fields). If we want to include them in the ECS component template, the detection rule build alert logic needs to be updated to not copy these fields from the source index to the alerts index, otherwise there will be many errors indexing.

cc @kqualters-elastic

[ymao1]

It was also discussed that we could include these fields but change the mapping to keyword instead of constant_keyword but diverging from the ECS mapping like this would show up in the data quality dashboard

cc @kqualters-elastic

[yctercero]

Thanks for the background @ymao1. Apologies if I missed these convos earlier. I'll bring it up at our Advanced Correlation meeting to see what we want to do here.

cc @paulewing @peluja1012 @marshallmain

[fitz003]

@yctercero I was curious if there was any update on this issue?

[LaZyDK]

Any updates?

[MSSP-BLKing]

As an MSSP, we use Elastic to monitor our clients environments. We have 1 use cases where having the data_stream.namespace mapped are quite important to us.

We measure and report on key metrics to each client, like number of alerts, number of closed alerts, number of alerts in progress. Unless I hack the index template, and replace it every upgrade, I can't separate this data amongst data_stream.namespace.

With our setup there's no reason to duplicate the SIEM rules for other spaces, I'm assuming that might be the other option. But adding the mapping to the index template works. Having an Alerts@Custom index component template would work. Seems to be an easy to implement solution.

Other uses for this could be filtering the alerts console for a specific client, this would especially be useful during an active cybersecurity event, where other clients who are physically separated from the client under attack are a lesser priority, and their alerts would only serve to confuse the incident response team.

This could also be helpful in workload balance amongst SOC analyst. As they are sometimes assigned to a client and not to the whole repository.

I'm not familiar enough with the deployment code and process from Github out to the production environments. But, at a bare minimum, and maybe as an additional request, deploying an Alerts@Custom index component template would allow adding mapping for this and other fields. And while you have your code cracked open, adding an Alerts@custom ingest processor would allow us to add an enrichment processor to our Alerts Index. I use this one to get the Max vulnerability priority​ rating for the host to the alert index. It adds an additional level of risk knowledge for the analyst.

[DigitalKraken]

So we found another workaround for this that prevents you from having to modify a component template and re-index. This also means you don't have to apply the fix every time you upgrade the cluster. You can filter on 'kibana.alert.ancestors.index : *-NAMESPACE-*' (replace NAMESPACE with your namespace obviously). We have verified that this works.

[LaZyDK]

So we found another workaround for this that prevents you from having to modify a component template and re-index. This also means you don't have to apply the fix every time you upgrade the cluster. You can filter on 'kibana.alert.ancestors.index : -NAMESPACE-' (replace NAMESPACE with your namespace obviously). We have verified that this works.

This is a nice workaround. It works on 8.17.0.

But we would still like the data_stream.namespace field to be present going forward, Elastic ;)

[MSSP-BLKing]

So we found another workaround for this that prevents you from having to modify a component template and re-index. This also means you don't have to apply the fix every time you upgrade the cluster. You can filter on 'kibana.alert.ancestors.index : -NAMESPACE-' (replace NAMESPACE with your namespace obviously). We have verified that this works.

This is a nice workaround. It works on 8.17.0.

But we would still like the data_stream.namespace field to be present going forward, Elastic ;)

kibana.alert.ancestors.index: *NAMESPACE*

I had to use the * instead of the _- and the -_ but it did work! I'm not sure if it'll work in my dashboard but it works from the security alert console on 8.17.0.

Very much concur, still need the data_stream.namespace option too.

[yctercero]

@LaZyDK @MSSP-BLKing we certainly hear you! :) Thank you for confirming the workaround.

We will look to prioritize this one and update when there's movement on our front. We appreciate your patience!