Review the fields of the wazuh-monitoring index

[Tostti]

Description

This PR aims to reduce the amount of data stored in the wazuh-monitoring index.
That index is only used for a visualization, which can be achieved with much less data, saving storage space and improving performance.

By default, every 15 minutes one log with the following structure is saved on the index for every agent:

{
  "_index": "wazuh-monitoring-2022.32w",
  "_type": "_doc",
  "_id": "j5flfIIBzH8x6PVkpeiR",
  "_version": 1,
  "_score": null,
  "_source": {
    "os": {
      "arch": "x86_64",
      "major": "2",
      "name": "Amazon Linux",
      "platform": "amzn",
      "uname": "Linux redacted |#1 SMP Tue May 7 02:26:40 UTC 2019 |x86_64",
      "version": "2"
    },
    "ip": "10.0.1.68",
    "mergedSum": "d9f3434a15b4ac879334867f1374da14",
    "status": "active",
    "manager": "wazuh-manager-master-0",
    "version": "Wazuh v4.3.6",
    "registerIP": "any",
    "configSum": "8361b0d0c5021ec104b85cf76b3ac0a0",
    "group": [
      "default"
    ],
    "dateAdd": "2022-07-20T09:23:14Z",
    "id": "001",
    "node_name": "master",
    "name": "Amazon",
    "lastKeepAlive": "2022-08-08T09:59:57Z",
    "timestamp": "2022-08-08T10:00:01.167Z",
    "host": "wazuh-manager-master-0",
    "cluster": {
      "name": "wazuh2"
    }
  },
  "fields": {
    "dateAdd": [
      "2022-07-20T09:23:14.000Z"
    ],
    "lastKeepAlive": [
      "2022-08-08T09:59:57.000Z"
    ],
    "timestamp": [
      "2022-08-08T10:00:01.167Z"
    ]
  },
  "sort": [
    1659952801167
  ]
}

With the proposed changes, only one log will be saved regardless of the number of agents, with the following structure:

 {
  "_index": "wazuh-monitoring-2022.48w",
  "_type": "_doc",
  "_id": "fin0xIQBU6oeV1BPEezV",
  "_version": 1,
  "_score": null,
  "_source": {
    "active": 3,
    "disconnected": 1,
    "never_connected": 0,
    "pending": 0,
    "total": 4,
    "timestamp": "2022-11-29T19:54:00.533Z",
    "cluster": {
      "name": "wazuh"
    }
  },
  "fields": {
    "timestamp": [
      "2022-11-29T19:54:00.533Z"
    ]
  },
  "sort": [
    1669751640533
  ]
}

To achieve that, instead of fetching all the agents and storing their information to verify the status, we fetch the strictly required fields for the visualization from the /agents/summary/status endpoint.

This allows us to reduce the amount of space required and the number of requests made to the API considerably, especially in environments with a high number of agents.

Issues Resolved

#4388

Evidence

After the changes, the visualization works in the same way.

Test

To test this PR, a valid response from the /agents/summary/status endpoint is needed. In case of using the Docker development environments and imposter, the changes from #4934 are needed.

It is also recommended to modify the frequency of the Task:monitoring, in Wazuh/Settings/Configurations to a lower number (in seconds, minimum is 60), to allow more logs coming to the index in less time (by default is every 15 minutes).

Scenario: have a Wazuh environment with at least one agent and wait for the necessary time to create several logs
When the user navigates to the Agents tab
Then the evolution visualization on the top-right side of the window should be displayed correctly.

Scenario: have a Wazuh environment with at least one agent and wait for the necessary time to create several logs
When the user adds or disconnects an agent, wait for the time to create a new log and navigates to the Agents tab
Then the evolution visualization on the top-right side of the window should display that change correctly.

*Note: if using imposter, the adding and disconnecting of the agents can be done modifying the file with the response and restarting imposter.

Check List

• All tests pass
  • yarn test:jest
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[Desvelao]

Thought: This approach is a breaking change. This is not backward compatible, so this means, the users after installing a plugin that contains these changes, could not see in the visualization of the plugin data related to a range of time previous to the plugin installation.

In the other hand, some users could have a use case that uses the current indexed data where other fields are used.

We should be sure of the consequences of these changes before integrating them.

Related issue #3625 to the issue that solves this pull request.

[asteriscos]

Missing changelog.
The code looks good. The visualization definition should be tested in Kibana 7.16.x and 7.17.x.

[github-actions]

Code coverage (Jest)	%	values
Statements	8.73%	( 3228 / 36972 )
Branches	4.46%	( 1281 / 28701 )
Functions	7.63%	( 700 / 9169 )
Lines	8.8%	( 3113 / 35391 )

[Desvelao]

thought: consider if the visualization of the agent status evolution is using the feature of allowed agents. If it does, then the suggested approach to index the computed data, will avoid the allowed agents feature can work. I am not sure if this has been taken into account.