[Security Solution][Data Views][Discuss] - Updating sourcerer/data view state management

[yctercero]

Purpose

The goal of this issue is to 1) collect documentation on sourcerer that help us understand the existing architecture 2) discuss updates needed to sourcerer.

Background

Security solution introduced data views into it's flows in 8.0 - the logic and component that control this are referred to as Sourcerer within the team. Security solution has a default data view it instantiates and uses that includes all relevant security solution indices -*elastic-cloud-logs-*,.alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*. Based on which page the user is navigating to, sourcerer decides which indices to filter on.

At a very very high level, sourcerer includes logic to:

• create the default data view
• poll to see when the alerts index is created, and once it has been created, updates the default data view to include it
• continuously check for any updates to the data view to refresh it
• state management of custom data view selection UI
  • This component allows a user to:
    • see which indices in a data view have data
    • select which indices from a data view to filter out (non persistent)

Over the course of moving to sourcerer we've come up against a number of performance issues. Additionally, there is a lack of understanding of what the best practice is for devs when fetching needed browser fields or index information resulting in use of very heavy/costly hooks throughout the code.

The first step in all of this was to establish an understanding of the existing architecture. Big kudos to @YulNaumenko who took this on and created some awesome flows for us.

Existing architecture

🔥 Diagram by @YulNaumenko

Takeaways from initial research

[Notes by @YulNaumenko]

• Simplify the set of the hooks to have responsible hook for a single task: create dataView + update Sourcerer store, fetch and cache IndexFields, update dataView
• Meeting with unified search team on the fieldsCaps and possibilities to create pre-build security persistent dataView
• Start building POC for replacing Sourcerer API and component with ad hoc data view
• Discuss the product feature of the non-persistent ad hoc dataviews for the security pages - move away from the complicated logic for timelines, detections, rest of the pages
• Approach indexFields fetch to have a paging there + figure out the ability to build the categories list by using the aggregations

Existing documentation

• Sourcerer component
• Sourcerer container
• Sourcerer redux
• Sourcerer api

Related issues

• [Security Solution] Data view per page not persisted  #136188
• [Security Solution] Remove sourcerer browser fields hover actions to help performance #131363
• [SecuritySolution][Field Browser] Improve filter selected performance #131530
• [Security Solution] Remove fields from sourcerer response #130917
• [Security Solution][Investigations] - Add pagination to alert details table #131358

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[yctercero]

After reviewing the diagram with Yuliia, these were some of the observations/questions I had:

• Can the default security solution data view be broken down into separate, more structured data views?
  • It currently includes the alerts index, which means users can't use this data view in rule creation without creating alerts on alerts
  • Separating out the alerts index would remove all the polling logic sourcerer currently has
• There's some additional complexity in that rules and exceptions continue to support both index patterns and data views
  • Diagram showed that we don't make any use of redux in the rules/exceptions flows and instead many of the individual components make expensive calls to get the relevant fields/index info. If a user is on a rule detail page, we should be able to leverage redux as the index patterns/data views aren't a field that a user would be changing frequently