[Update dashboards] Global vulnerabilities dashboards

[gdiazlo]

Introduction

With the new vulnerability detector work (wazuh/wazuh#14153), we can start implementing a global vulnerability dashboard which can manage all detected vulnerabilities across all endpoints.

We will implement this dashboard under Modules / Vulnerabilities:

without selecting an agent, this page will show the default dashboard, which will be designed like the following mock:

We won't be using the embedded Discover to implement the inventory or the vulnerabilities table, instead, we will use a standard table component with an expand action, which will display a fly-out with the details about the vulnerability selected. This design will take advantage of the new vulnerabilities index schema to ensure all vulnerabilities contain a minimum set of information required by our users.

The single agent dashboard will use the global one with an agent filter applied.

Development plan

In order to create the new vulnerabilities dashboard we need to define different tasks/steps:

• Research stage

  • [Vulnerabilities dashboard] Research about vulnerabilities dashboard context/scope #5800
  • [Vulnerabilities dashboard] Add POC to create a dashboard using osd artifacts #5839

• Components and services

  • [Vulnerabilities dashboards] Add search bar services/hooks #5945
    • Known issue solved: [Vulnerabilities dashboards] Wrong agent.id filters loaded in search bar by default #5969
  • [Vulnerabilities dashboards] Add dashboard services/hook #5946

• Demo stage

  • [Vulnerabilities dashboard] Create the Dashboard tab using OSD plugins #5894
  • [Vulnerabilities dashboard] Replace the Inventory tab using OSD plugins #5895

• Improve and fixes stage

  • [Vulnerability Dashboards] Fix vulnerability dashboard filters #6064

• Add vulnerabilities index patter validation configuration and health-check

  • Add vulnerabilities index patter validation to health-check #6146

• Fix show call-out warning when there is no data in the vulnerabilities index

  • Show call-out warning when there is no data in the vulnerabilities index #6166

• Fixed filter modal of bar chart of vulnerabilities dashboard

  • Fix filter modal of bar chart of vulnerabilities dashboard #6170

• Fix default filters per tab in Vulnerabilities module

  • Fix default filters per tab in Vulnerabilities module #6165

• **Remove Inventory table from vulnerability dashboard tab **

  • Remove table from vulnerability dashboard tab #6259

• Disable by default the vulnerabilities health-check

  • Disable by default the vulnerabilities health-check #6319

• Maximized visualizations don't display properly in vulnerabilities dashboard

  • Maximized visualizations don't display properly in vulnerabilities dashboard #6322

• Add pinned agent to vulnerability section

  • Add pinned agent to vulnerability section #6359

• Update vulnerability configuration block

  • Support new vulnerability configuration block #6370

• Create visualization definitions using the OSD Visualize plugin (2.9.0) and the new vulnerabilities index pattern.
• Use the dashboard plugin react component to render the vis definitions created in the previous step
• Add a searchbar component to filter the events
• Repeat the process for the dashboard and inventory tabs

• Test and documentation stage
  • Add unit test, e2e testing, etc.
  • Add documentation to reuse in other module dashboards

Plan

We want to implement a new dashboard tab:

• Vulnerabilities search bar
  • We must use the platform search bar component
  • We must search in the new vulnerability index
  • Each time we search, all the dashboard must update to reflect the results based on the search bar filter
• Cross search vulnerability with alerts
  • Investigate how can we cross-search the vulnerability index with the general alerts index to search for additional information from within the vulnerability dashboard (It's not possible cross-search the vulnerability index with the general alerts index)
• Filter by vulnerable product
  • A user must be able to quickly select a product from a list and add it as a filter to the search bar
• Filter by operating system version
  • A user must be able to quickly select an OS version from a list and add it as a filter to the search bar
• Easy filter by agent group
  • A user must be able to quickly select an endpoint group from a list and add it as a filter to the search bar
• cves by severity
  • A user must be able to quickly see the amount of CVEs by severity: critical, high, medium, low (There is not enough information in the index to make a graph like this. Replaced with Accumulation of the most detected vulnerabilities)
• open vs closed vulnerabilities over time
  • A user must be able to quickly see the tendency of open vs closes vulnerabilities. This information comes from the alerts index, instead of the vulnerabilities one, so this graph must have its own time filter
• top 10s graphs
  • A user must be able to see
    • the most detected vulnerabilities
    • the most vulnerable endpoints
    • ...
• vulnerabilities table
  • sort by values like the vulnerability score
  • actions:
    • ~~See details on a fly out ~~(It's not possible)
    • Go to alerts index with a filter by that vulnerability (It's not possible)
• evaluate the performance of the dashboard in scenarios with thousands of endpoints (does not apply to this issue)
• Reporting (this functionality will be implemented later)
  • A user should be able to export the query results
  • A user should be able to create a report out of the dashboard

We might need to implement new configuration options to support the new vulnerability detector under the current configuration module.

• New vulnerability detector configuration options

We want to implement a new inventory tab:

• Add related native searchbar
• New inventory table using native indexer table (as is possible)
• Add filters and columns switcher

Dependencies

This work depends on the vulnerability index schema and a set of sample data to build the queries to generate and test the dashboard visualizations and queries.

Related issues

We will add here issues from our backlog and projects related to this effort.

• Refine vulnerabilities dashboard #6533
• Redesign the Vulnerabilities dashboard visualization #6581
• [Vulnerabilities Inventory] Error when click on show all button in columns selector #6164
• Show call-out warning when there is no data in the vulnerabilities index #6166
• Add vulnerabilities index patter validation to health-check #6146
• Health-check error when there's no vulnerabilities indices #6209
• Fix responsive view on vulnerability dashboard #6245
• Disable by default the vulnerabilities health-check #6319
• Vulnerability detail truncated text in big fields #6332
• Refactor top N tables in vulnerabilities dashboard #6353
• Health-check vulnerabilities index-pattern validation should be removed #6395
• Add cluster filter by default in vulnerabilities dashboard and inventory tab #6384
• Add validations in the Vulnerabilities dashboard and inventory views  #6394
• Add manager name to filter vulnerabilities alerts with cluster disabled #6407
• Use agent.name in vulnerability dashboard table #6419
• Change most vulnerable endpoints label to endpoint name #6416
• Filter agent.id 000 in vulnerabilities module #6420
• Visualization filters are lost when changing tabs in Vulnerabilities module #6447
• Change visualizations in vulnerabilities dashboard #6485
• Add JSON visualization to Vulnerability Detection inventory flyout #6589
• Add wildcard to the default vulnerabilities index pattern #6641
• Change the Wazuh server filter field to wazuh.cluster.name in vulnerabilities #6650
• Vulnerabilities inventory table truncated #6645
• Inconsistent wazuh.cluster.name filter #6654
• The vulnerability detector event filter can sometimes be removed, displaying confusing information #6316
• Vulnerabilities inventory tab table doesn't maximize properly #6323
• Remove scroll from vulnerabilities inventory table #6324
• [Vulnerabilities dashboards] Search error when results exceed 10000 elements #6097
• Change Vulnerabilities Inventory flyout title text #6661

[sempervictus]

Reporting and export of query results for this dashboard might get interesting based on the proposed composition. Suggest adding it as a first tier specification to avoid complications formatting those down the line.
Thanks for picking this up folks.

[chantal-kelm]

Based on the details of the topic, we elaborated the following dasboards, we will continue to work on their appearance and on the data to be displayed.

[chantal-kelm]

Update dashboard

[jbiset]

Guide to generate random data for the vulnerabilities module

To test the new vulnerabilities module, it is necessary to generate data from the new wazuh-states-vulnerabilities vulnerability index in the indexer.

Generation of vulnerability index data

Having the environment up, use the random data generation script to generate the data as shown in the following image.

NOTE: keep in mind that the indexer port (9200) has to be exposed in the container configuration.

Creation of the `wazuh-states-vulnerabilities` index

Go to Dashboards Managements and create the corresponding `wazuh-states-vulnerabilities` index as shown in the following images:

NOTE: It is important to specify the Custom index pattern ID with wazuh-states-vulnerabilities so that a ramdom ID of the index is not automatically generated.

Mapping the index fields

Go to Dev tools and insert the field mapping through a PUT, then do a GET of some field to test.

At this point it is necessary to do a DELETE as shown in the image and repeat again only the data generation but with a greater number of data.

Dev Tools code

GET wazuh-states-vulnerabilities/_mapping/field/agent.name

PUT _index_template/wazuh-vulnerability-detector
{
  "index_patterns": [
    "wazuh-states-vulnerabilities"
  ],
  "priority": 1,
  "template": {
    "mappings": {
      "date_detection": false,
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "build": {
              "properties": {
                "original": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "ephemeral_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "action": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "agent_id_status": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "category": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "code": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "created": {
              "type": "date"
            },
            "dataset": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "duration": {
              "type": "long"
            },
            "end": {
              "type": "date"
            },
            "hash": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ingested": {
              "type": "date"
            },
            "kind": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "module": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "original": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            },
            "outcome": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "provider": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "reason": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "reference": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "risk_score": {
              "type": "float"
            },
            "risk_score_norm": {
              "type": "float"
            },
            "sequence": {
              "type": "long"
            },
            "severity": {
              "type": "long"
            },
            "start": {
              "type": "date"
            },
            "timezone": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "url": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "host": {
          "properties": {
            "os": {
              "properties": {
                "family": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "full": {
                  "fields": {
                    "text": {
                      "type": "text"
                    }
                  },
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "kernel": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "name": {
                  "fields": {
                    "text": {
                      "type": "text"
                    }
                  },
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "message": {
          "type": "text"
        },
        "package": {
          "properties": {
            "architecture": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "build_version": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "checksum": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "description": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "install_scope": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "installed": {
              "type": "date"
            },
            "license": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "reference": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "size": {
              "type": "long"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "tags": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "vulnerability": {
          "properties": {
            "category": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "classification": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "description": {
              "fields": {
                "text": {
                  "type": "text"
                }
              },
              "ignore_above": 1024,
              "type": "keyword"
            },
            "enumeration": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "reference": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "report_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "scanner": {
              "properties": {
                "vendor": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "score": {
              "properties": {
                "base": {
                  "type": "float"
                },
                "environmental": {
                  "type": "float"
                },
                "temporal": {
                  "type": "float"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "severity": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "codec": "best_compression",
        "mapping": {
          "total_fields": {
            "limit": 1000
          }
        },
        "refresh_interval": "2s"
      }
    }
  }
}

DELETE wazuh-states-vulnerabilities

Finally it is necessary to refresh the index in Dashboard Management

[micsinyei]

I am very interested in this dashboard, how do you show the active vulnerabilities since the current filtering does not allow that?

Can you provide some information on it?

[botkillern]

Creation of the wazuh-states-vulnerabilities index

I think the comment from @jbiset is about it. (Creation of the wazuh-states-vulnerabilities index)
You need to create a new index just for vulnerabilities...

Anyway, still in development.

I'm very interested in this feature too.

[Machi3mfl]

@botkillern and @micsinyei, thanks for your feedback.
That's right, @botkillern. This is only for development purposes.

[Machi3mfl]

Vulnerabilities inventory

[ygeorgiev]

do you have ETA?

[Dretreyt]

Update dashboard

Hi,

Have you got the guides on setting up these visualizations please?

Thanks

Rob

[Jarli01]

Yeah this would be amazing if released, having a default overview of all vulnerabilities out of the gate would simply make the tool useful.

Individual results, while useful is annoying to have to go back and forth between.

[burdorff]

If I'm reading all the details here correctly, this is in process for Wazuh 4.8 and is present in the current beta. It appears that the primary delay was the need for a new indexer/index to keep track of the vulnerabilities that are "current" instead of listing both active and solved as the Events table does.

[Jarli01]

@burdorff correct, the need to generate a new indexer appears to be the hold up.

[mesutgungor]

Mean Time To Detect MTTD and Mean Time to Fix MTTF are the two crucial KPIs to put this dashboard I think.