New vulnerabilities index

[gdiazlo]

Introduction

With the new vulnerability detector work (wazuh/wazuh#14153), we are going to create new indices which will contain the information about all active vulnerabilities in all endpoints across a Wazuh environment.

Design

• The index won't have a timestamp, as it will represent the current global status of all vulnerabilities found in all endpoints
• The index will contain only ECS fields, and will reject any non-mapped fields with an error
• The index will have the necessary settings to ensure the new dashboard can work without any performance errors

We also need to investigate a way for our indexer package to deploy the indexes and their mappings on a fresh installation, and upgrade them on upgrades.

To test the performance, we will use a canonical environment as described in our documentation.

Plan

• Create vulnerabilities index mapping #6
• Create a sample dataset for testing
• Vulnerability detection state index stress testing #31
• Define default search fields
• Create the installation/update procedure

Related issues

We will include here related issues outside our plan which might be affected by this development.

[AlexRuiz7]

Task: Create a sample dataset for testing

Done in #6 (comment)

[AlexRuiz7]

Define default search fields

We are using these ECS fields in the index:

• base
• agent
• ecs
• event
• host.os
• package
• vulnerability

I suggest adding at least one property of each field to the default search fields of the index. We need to have in consideration the significance of these properties, as well as their type, as they can have impact on the searched performed afterward. For example:

A keyword field type contains a string that is not analyzed. It allows only exact, case-sensitive matches.
Source: OpenSearch's Keyword field type documentation

These could be:

Field	Type
base.tags	keyword[ ]
agent.id	keyword
ecs.version	keyword
event.id	keyword
event.module	keyword
event.severity	long
host.os.family	keyword
host.os.full.text	text
host.os.version	keyword
package.name	keyword
package.version	keyword
vulnerability.id	keyword
vulnerability.description.text	text
vulnerability.severity	keyword

which translates into the following lines of code that have to be added to the index template:

"settings": {
  "index.query.default_field": [
    "base.tags",
    "agent.id",
    "ecs.version",
    "event.id",
    "event.module",
    "event.severity",
    "host.os.family",
    "host.os.full.text",
    "host.os.version",
    "package.name",
    "package.version",
    "vulnerability.id",
    "vulnerability.description.text",
    "vulnerability.severity",
  ]
}

Resources:

• https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html
• https://codecurated.com/blog/basics-of-elasticsearch-for-developer/
• https://codecurated.com/blog/elasticsearch-text-vs-keyword/

[AlexRuiz7]

Create the installation/update procedure

@wazuh/security-warriors will be in charge of this.

[AlexRuiz7]

I tested again in a fresh environment and the mappings work as expected.