Remove embedded discover

[gdiazlo]

Description

We want to remove the embedded discover and replace it with a more modern component. This will allow us to remove angular as a dependency, improve usability and performance. Also, this will allow us to adapt to the upcoming changes from the upstream OpenSearch related to the new discover, the changes of the OUI library and others.

We use discover in multiple places, the most apparent one is the Events tabs:

But its components are also used to build other visualizations, managing filters and some events. The removal of the discover will require a refactor of these uses.

Our strategy will be to create our own component based on the one we already created for the new inventory tab:

Screen.Recording.2023-10-25.at.10.52.34.mov

We will add to the current inventory tab the time filters and the timeline visualization as shown in the discover.

In general our implementation strategy will be to add the necessary changes to make the discover redundant, and then, remove it. Also, we don't want to lose functionality, if there are use cases not identified in this issue, discuss them in the comments sections below. We will update the issue body with the conclusion.

Functional requirements

As a user, I want to be able to:

Explore data:

• view the events in a timeline visualization.
• customize the fields displayed in the table
• receive events as they are produced (auto reload)
• see the whole event details in a fly-out panel

Generate reports:

• manage saved queries (CRUD)
• generate a report in the reporting plugin, based on a query
• download a query as a CSV.

Explore event context:

• click on a scripted field and follow a link to other modules
• click a field and display a fly-out with a sub-query of related events

Non-functional requirements

• Research the use of scripted fields to add context to values anywhere in our data tables

Implementation restrictions

• Use the newly developed inventory component, adapting or increasing its features. These modifications should not break the current behaviors.

Plan

• [Remove discover] Add all the necessary components to compose the new discover #6197
• [Remove discover] Replace discover children components in vulnerabilities modules tab #6222
• [Remove discover] Replace discover on threat hunting module #6236
  • Create histogram visualization
  • Enhance column rendering to support custom call to actions
  • Develop the Save Query feature
• Implement the new discover view in the rest of the modules
  • [Remove discover] Replace discover on vulnerability detection module #6273
  • [Remove discover] Replace discover on configuration assessment module #6276
  • [Remove discover] Replace discover on file integrity monitoring module #6277
  • [Remove discover] Replace discover on Docker module #6278
  • [Remove discover] Replace discover on AWS module #6279
  • [Remove discover] Replace discover on Google cloud module #6280
  • [Remove discover] Replace discover on Github module #6281
  • [Remove discover] Replace discover on Office 365 module #6282
  • [Remove discover] Replace discover on PCI DSS module #6295
  • [Remove discover] Replace discover on GDPR module #6296
  • [Remove discover] Replace discover on HIPAA module #6298
  • [Remove discover] Replace discover on NIST 800-53 module #6299
  • [Remove discover] Replace discover on TSC module #6300
  • [Remove discover] Replace discover on MITRE ATT&CK module #6450
  • [Remove discover] Replace discover on Virustotal module #6451
  • [Remove discover] Replace discover on Malware Detection module #6452
• [Remove discover] Refactor modules dashboards with new embeddable component #6477
• Standardize filter management mechanisms #6499
  • Pinned agent management refactor #6644
• [Remove discover] Implement new PDF reports in dashboards #6556
• [Remove discover] Refactor angularJS controllers #6535
• [Remove discover] Migrate AngularJS routing to React #6680

Related fixes/features

• [Remove discover] Change index pattern selector mechanism #6434
• Fix maximized visualizations styles #6865
• Replace common components vuls module #6235
• Replace discover threat hunting #6254
• Replace discover Configuration assessment #6285
• Replace discover AWS #6288
• Replace discover Office 365 #6290
• Replace discover Google cloud #6289
• Replace discover Docker #6286
• Replace discover Vulnerability #6275
• Replace discover file integrity monitoring #6287
• Replace discover PCI DSS, GDPR, HIPAA, TSC and NIST 800-53 modules  #6297
• Replace discover Mitre Attack, Virustotal and Malware detection #6459
• Implement embeddable dashboard on PCI DSS #6504
• Adapt dataSource to Configuration assessment #6649
• Implement embeddable dashboard on GDPR #6506
• Implement embeddable dashboard on GitHub #6537
• Implement embeddable dashboard on Office 365 #6528
• Fix cluster status response #6675
• Fix filters are cleaned when navigate throw the modules #6674
• Adapt the reporting to the rendering of dashboards #6558
• Fix remove filter icon visibility when pin and unpin agent #6685
• Fix discover flyout and improve events columns #6691
• Fix modules redirects with pre-defined filters #6712
• Fixed pinned agent filter when unpin and change pinned agent #6734
• Use index pattern id instead of title #6746
• Rendering changes to embedded dashboards #6752
• Replace discover table on FIM inventory flyout #6753
• Remove empty panel when there is no data #6756
• Fix wrong cleaned filters when browser back button is triggered  #6771
• Timestamp data add in tables of Events tab #6792
• Some links in the threat hunting dashboard table do not redirect or have no links #6845
• Add colours to GitHub dashboard metric visualisations #6857
• The selected filter is not applied when redirecting to the events tab in the file integrity monitoring module #6847
• Unification the rendering of flyout columns #6848
• Add field links in discover document details #6843
• The dashboard of the mitre module is the same with or without a pinned agent #6878
• Fixed render bug in security alerts table #6883
• Add tooltips to new discover table call to actions #6889
• Fix error generating dashboard reports on Firefox #6902
• Add default sorting and columns in all data grid tables #6912
• Fix events data table pagination exception #6917
• Sync initial filters with app state #6927
• Use absolute date range on change page requests #6923
• Keep the appState filters on refreshing page #6950

[Machi3mfl]

New Discover

Components features

Search bar

• Search input to filter by a keyword
• Date picker to filter by a date (range, absolute, etc)
• Add filter modal to filter using UI

Histogram chart

• Hits chart by time
• Visualization options (Maximize and inspect chart)

Data grid

• Table with pagination and rows per page selector
• Table with headers features like sort, columns selector, export results to CSV, density layout selector, etc
• Custom column rendering (Is possible to render any component for a specific column via code)

type tDataGridColumn = {
    render?: (value: any) => string | React.ReactNode;
} & EuiDataGridColumn;

export const discoverModuleColumns: tDataGridColumn[] = [{
    id: 'timestamp',
},
{
    id: 'agent.name',
    render: (value: string) => { // return any React.ReactNode for the cell
        return (
            <EuiBadge>{value}</EuiBadge>
        )
    }
},
{
    id: 'rule.description'
},
{
    id: 'rule.level'
},
{
    id: 'rule.id'
}]

• Row details flyout to show all the document data

[asteriscos]

I will close this issue, because the angularJS routing will be solved in the next stage of the release.

• [Remove discover] Migrate AngularJS routing to React #6680

[Desvelao]

Added pull request to the list of fixes: #6950