Use elasticsearch index.auto_expand_replicas for wazuh app indices

[epol]

Describe the solution you'd like
I think it would be nice if by default service indices (wazuh-monitoring-3.x-*, .wazuh and .wazuh-version) are created with the setting index.auto_expand_replicas set to 0-1. This automates the expansion of the number of replicas of a index as a function of the number of data nodes in a cluster.

This would on one hand ensure data availability in case of a multi (data) node setup while keeping the cluster state green in single node setups.

Another possibility would be to use the value 0-all at least for the .wazuh index (since it contains the API credentials) in order to ensure the best availability. I leave to you the evaluation of this possibility.

Describe alternatives you've considered
Configuring the config.yml file is for sure a valid alternative, but it may not be very straightfoward for new users and the default replica value is 0 (that may cause some issue when a data node fails).

Additional context
Elasticsearch documentation for the requested feature: Index modules. Browsing the older releases you can see that it was already supported in Elasticsearch 6.8 (so it can be implemented in both versions of the plugin)

[jesusgn90]

Hi @epol ,

You are fully right, however, we are working on a new way to manage the data from .wazuh and .wazuh-version indices.

Next versions will do a soft deprecation for those indices, instead, the app will use a registry file for storing the information of .wazuh-version index, then Kibana will manage this file and that index will be no longer needed.

For the .wazuh index, we are planning to dump it to our config.yml file, it's also a soft deprecation so far, but this index is also going to disappear in further versions.

For the wazuh-monitoring-3.x- indices you are right too, and we should add it in the monitoring template, which lives in the Wazuh app (see file). In addition, we may want to add optional settings to our config.yml for this too, so the user can change it without touching the template.

Regards,
Jesús

[epol]

Hi @jesusgn90 ,

thank you for your feedback.

Do you have any pointer to this new way to manage data? (maybe an issue)

What I would like to understand is how would this interact with multiple instances of kibana running on different servers, with the current configuration all instances would share the .wazuh index (but non the config.yml file) making configuration syncronization automatic.

Regards,
Enrico

[jesusgn90]

Hi @epol,

Do you have any pointer to this new way to manage data? (maybe an issue)

Sure, take a look at #1465 which is still under development, and #1467 which is closed now (we've added more commits after closing it because we've found some errors testing it).

What I would like to understand is how would this interact with multiple instances of Kibana running on different servers

Yes, that's a good question. We did something similar in the past for the monitoring indices as you can see here. That's said, we may want to replicate a similar logic for this use case and we'll study it.

Regards,
Jesús

[juankaromo]

Hi @epol,

Since the last release of Wazuh 3.11.0, the .wazuh index has been deprecated in favor of the wazuh.yml configuration file, and .wazuh-version index was also previously deprecated in favor of the wazuh-registry.json file, so this problem no longer applies.

Finally, we still have to add optional settings to our wazuh.yml to configure monitoring, and that the user can change it without touching the template, but this is still under study.

If you have any other questions, problems or suggestions do not hesitate to open a new issue. I'm closing this one for now.

Regards,