[Security Solution] Data view per page not persisted

[mjraa]

Describe the bug:
When changing the data view in a page, it also changes the data view in other pages. See video for reference.

I am unsure if this is by design, but this seems problematic because users will create a data view with all relevant index patterns for all pages. This creates unnecessary load (hitting irrelevant indices (across all tiers), field caps API executed for all index patterns, etc).

data_view_not_persisted_between_pages.mp4

Kibana/Elasticsearch Stack version:
8.3.1

Server OS version:
Linux (running in Kubernetes)

Browser and Browser OS versions:
All

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.):
ECK downloading the containers from an internal registry

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):
Security app, at least Overview, Hosts, Network and Users pages

Steps to reproduce:

1. Create two data views
2. In the Hosts page, for example, select the first data view
3. Access the Network page, for example, change the data view to the second one
4. Access the Hosts again and see that the data view has been changed to the second one

Current behavior:
Data view per page not persisted

Expected behavior:
Data view per page persisted

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[yctercero]

cc @peluja1012 @jethr0null

@mjraa I am not sure if this was by design. @stephmilovic - do you know if the persistence of a selected data view was by design?

@mjraa I do know that the intention of creating the default security data view was to have all the security indices together to be able to add runtime fields across them. Then based on what page you're on, sourcerer decides which indices to select from that data view. It does in fact create some performance issues as you noted it might.

[stephmilovic]

yes it was by design.

[mjraa]

Hi @yctercero @stephmilovic ,

Can you please explain the design and tradeoffs considered?

I think specially the SIEM use case typically requires high-volume data sources. Querying all SIEM relevant indices, even when not needed for a given page, may work in a small cluster, but not in a big one.

[yctercero]

@mjraa I can't speak to the original design tradeoffs considered, but I can say that we are actively examining the design to hone in on performance improvements. I believe that part of the intent in having the default elastic security data view that encompasses all indices was to make it easier for junior analysts to interact with the application as, under the hood, sourcerer (the logic behind the data view picker) self-filters which indices to use for each page.

@paulewing may be able to give a bit more historical context here?

If you wouldn't mind sharing - what is the behavior that you would find most useful? Would having the ability to set a default data view to query for each page be helpful? Do you ever use the default security solution data view in your flows?

[mjraa]

@yctercero Thank you very much for this!

In my current organization we are using the Security solution. We have yearly data retention requirements and, due to high-volume nature of some data sources, this results in a fairly large cluster(s) with many indices/shards (we also bought the enterprise license to benefit from the frozen tier, for more cost effective solutions).

Due to the high number of indices/shards, we find important to optimize multiple things, including what is queried. Querying everything results in a big overhead affecting the cluster stability.

We would, therefore, like to define a data view per page. That way, we would make sure to only query relevant data.

I understand the user experience angle. For us, it is not worth it because, besides the big query overhead (the most important aspect), a junior analyst can still choose what data to query by reading the index names in the data view (in our case, we create our own data views, because we use custom index names, maybe except for winlogbeat-*).

Furthermore, persisting the data view per page should not invalidate the first design - it is still possible to choose the same one across all pages.

[yctercero]

@mjraa thank you, this is really great feedback. I agree that one definitely does not invalidate the other.

As I mentioned - we're actively examining the design here and I will definitely bring this feedback up. I'll be sure to link to this ticket as well once we kick things off.

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)