SPIKE - Installation assistant

[teddytpc1]

Objective
https://github.com/wazuh/internal-devel-requests/issues/1319

Description

As part of the DevOps overhaul objective, we need to research, analyze alternatives, and design how to implement the following changes:

1. Simplification:

   • The installation assistant must be streamlined.
   • Remove the offline installation option from the script.
   • Eliminate additional configuration options to reduce complexity.

2. Package Usage:

   • Transition to using Wazuh packages instead of package repositories.

3. Configuration Standardization:

   • Utilize out-of-the-box Wazuh configurations for installations.

4. Requirements Validation:

   • Add a validation step to ensure minimum requirements are met before proceeding with the installation.

5. Tool Optimization:

   • Leverage well-known tools to improve reliability and maintainability.
   • Avoid using additional dependencies to the ones required by Wazuh.

6. Testing Improvements:

   • Enhance the DevOps-owned testing for the installation assistant.
   • The GH Workflows must allow testing with Production, pre-release, and development packages.

Implementation restrictions

• Testing Environment:
  • Tests must be implemented using GitHub Actions (GHA).
• Compatibility:
  • Ensure compatibility with supported operating systems and environments.
• Minimal Maintenance:
  • Focus on reducing complexity and ensuring the script is easy to maintain.

Plan

1. Research & Analysis:

   • Review the current installation assistant to identify areas of complexity.
   • Analyze the impact of removing offline installation and additional configuration options.

2. Simplification:

   • Remove the offline installation option from the script.
   • Eliminate unnecessary configuration options to streamline the process.

3. Package & Configuration Updates:

   • Modify the script to use Wazuh packages instead of package repositories.
   • Ensure out-of-the-box configurations are implemented throughout the installation process.

4. Requirements Validation:

   • Add pre-installation checks to validate system requirements.
   • Notify users if minimum requirements are not met and provide guidance.

5. Tool Integration:

   • Replace custom implementations with well-known tools where applicable.
   • Document the reasons for tool selection and usage.

6. Testing Enhancements:

   • Create comprehensive GHA workflows to test the installation assistant.
   • Include tests for package installation, configuration application, and requirements validation.

7. Documentation Updates:

   • Update the installation assistant documentation to reflect the new process.
   • Simplify instructions and focus on the prerequisites for using the script.

8. Validation & Feedback:

   • Test the updated installation assistant across various environments.
   • Collect feedback from stakeholders and users to ensure alignment with expectations.

[Enaraque]

Update report

The different parts of the code have been reviewed to assess the feasibility of using Python as the programming language, particularly concerning the external dependencies it may require. This investigation is tentative for now, as the installation assistant for version 5.0.0 will differ significantly from what we currently know. Therefore, to conduct an investigation that allows us to determine if Python is the right language and whether implementation will not encounter too many external dependencies, it will be divided into three parts:

• Core Code: This is the most relevant part of the investigation. It represents the code that is most likely to remain in the implementation, in other words, the code that is most likely to be part of the new implementation in Python.
• Adaptable Code: This includes probable code to be implemented but might undergo changes in the Python implementation or due to future decisions.
• Deprecated Code: This consists of code likely to be removed from the installation assistant.

This division in the investigation reflects the priority of the different parts, with the Core Code being the most critical for decision-making and the Deprecated Code being the least significant.

[Enaraque]

Dependencies of the Installation Assistant

Types of Dependencies

The installation assistant has two types of dependencies:

1. Libraries required for the proper functioning of Wazuh: These libraries are not tied to the installation assistant's code but to Wazuh's main components, so they do not impact this investigation.

These libraries are:

Components	Libraries
Wazuh Indexer	- apt: debconf, adduser, procps - yum: coreutils
Wazuh Manager	- apt: gnupg, apt-transport-https
Wazuh Dashboard	- apt: debhelper, tar, curl, libcap2-bin, - yum: libcap

2. Libraries used by the code: These are libraries necessary for the execution of the installation assistant's code. They may or may not be required for Wazuh's proper functioning. These are the ones analyzed in this investigation.

Library	Description	Python Library	Native in Python	Minimum Python Version
grep	Text search utility	re	Yes 🟢	1.x
tar	Archive compression/decompression tool	tarfile	Yes 🟢	2.3
coreutils	Basic system utilities	os, pathlib	Yes 🟢	3.4 (pathlib)
procps-ng	Used to manage processes (e.g., kill)	os, signal	Yes 🟢	1.x
gawk	Text processing utility	re	Yes	1.x
curl	Data transfer utility	requests, urllib	Requests: No 🔴, urllib: Yes 🟢	3.0 (urllib)

Minimum Python version to use

Most of the libraries to be used have been part of Python for quite some time. However, the minimum version required, for now, would be 3.4, as pathlib is included starting from this version.

Important

It is also important to note that elements such as dataclasses (>=3.7), f-strings (>=3.6), or Enums (>=3.4) can be used in the code logic. Therefore, a definitive minimum Python version to use has not yet been established.

Conclusion

After completing the investigation, it can be concluded that all dependencies of the installation assistant are native to Python, so no additional dependencies need to be installed for its proper functioning.

Note

The most commonly used library for making HTTP requests in Python is requests. However, it is not necessary to install it because urllib is a native Python library and can be used for the same purpose. Still, if more complex HTTP requests are required in the future, the possibility of needing to install requests should not be ruled out.

Disclaimer

As shown, there are currently no external libraries required in the installation assistant's code. However, this does not mean that external libraries won't need to be added as the code evolves, such as the use of Pydantic, among others.

[Enaraque]

Update report

First, it is necessary to specify which options from the installation assistant will be kept and which will be deprecated.

For now, it is clear that both offline installation and the password-tool will be deprecated starting from version 5.0.0.
The cert-tool will remain for certificate generation.

Below is a list of all the options currently available in the installation assistant. Each option will have an associated color tag indicating whether it will be maintained or deprecated in version 5.0.

Options List

Legend:

🔴 Deprecated option
🟡 Option under review
🟢 Maintained option

Option	Description	Status
-a, --all-in-one	Install and configure Wazuh server, Wazuh indexer, Wazuh dashboard.	🟢
-c, --config-file <path-to-config-yml>	Path to the configuration file used to generate wazuh-install-files.tar file containing the files that will be needed for installation. By default, the Wazuh installation assistant will search for a file named config.yml in the same path as the script.	🟡
-d [pre-release|staging], --development	Use development repositories. By default it uses the pre-release package repository. If staging is specified, it will use that repository.	🟡
-dw, --download-wazuh <deb|rpm>	Download all the packages necessary for offline installation. Type of packages to download for offline installation (rpm, deb)	🔴
-fd, --force-install-dashboard	Force Wazuh dashboard installation to continue even when it is not capable of connecting to the Wazuh indexer.	🟡
-g, --generate-config-files	Generate wazuh-install-files.tar file containing the files that will be needed for installation from config.yml. In distributed deployments you will need to copy this file to all hosts.	🟢
-h, --help	Display this help and exit.	🟢
-i, --ignore-check	Ignore the check for minimum hardware requirements.	🟢
-id, --install-dependencies	Installs automatically the necessary dependencies for the installation.	🟡
-o, --overwrite	Overwrites previously installed components. This will erase all the existing configuration and data.	🟢
-of, --offline-installation	Perform an offline installation. This option must be used with -a, -ws, -s, -wi, or -wd.	🔴
-p, --port	Specifies the Wazuh web user interface port. By default is the 443 TCP port. Recommended ports are: 8443, 8444, 8080, 8888, 9000.	🟡
-s, --start-cluster	Initialize Wazuh indexer cluster security settings.	🟡
-t, --tar <path-to-certs-tar>	Path to tar file containing certificate files. By default, the Wazuh installation assistant will search for a file named wazuh-install-files.tar in the same path as the script.	🟡
-u, --uninstall	Uninstalls all Wazuh components. This will erase all the existing configuration and data.	🟢
-v, --verbose	Shows the complete installation output.	🟢
-V, --version	Shows the version of the script and Wazuh packages.	🟢
-wd, --wazuh-dashboard <dashboard-node-name>	Install and configure Wazuh dashboard, used for distributed deployments.	🟡
-wi, --wazuh-indexer <indexer-node-name>	Install and configure Wazuh indexer, used for distributed deployments.	🟡
-ws, --wazuh-server <server-node-name>	Install and configure Wazuh manager and Filebeat, used for distributed deployments.	🟡

Questions to Discuss

Based on this table, several questions arise for discussion with the team members:

• Will the password-tool be maintained?
• Will the cert-tool retain its current options, or will it be simplified?
• Will it be possible to use a custom directory for both config.yml and the tar with the certificates, rather than requiring them to be in the same path as the wazuh-install? (Regarding the -c and -t tags)
• Will the development mode installation still be used? (Tag -d)
• Will customizing the dashboard port still be supported? (Tag -p)
• Without using Filebeat for communication between the server and indexer, how will it be possible to verify that the communication between them is working correctly?
• Will still be possible to do the distributed installation or only the AIO?
• Will the start-cluster option still be available?

[c-bordon]

Options List

Legend:

🔴 Deprecated option
🟡 Option under review
🟢 Maintained option

Option	Description	Status
-a, --all-in-one	Install and configure Wazuh server, Wazuh indexer, Wazuh dashboard.	🟢
-c, --config-file <path-to-config-yml>	Path to the configuration file used to generate wazuh-install-files.tar file containing the files that will be needed for installation. By default, the Wazuh installation assistant will search for a file named config.yml in the same path as the script.	🟡
-d [pre-release|staging], --development	Use development repositories. By default it uses the pre-release package repository. If staging is specified, it will use that repository.	🔴
-dw, --download-wazuh <deb|rpm>	Download all the packages necessary for offline installation. Type of packages to download for offline installation (rpm, deb)	🔴
-fd, --force-install-dashboard	Force Wazuh dashboard installation to continue even when it is not capable of connecting to the Wazuh indexer.	🟡
-g, --generate-config-files	Generate wazuh-install-files.tar file containing the files that will be needed for installation from config.yml. In distributed deployments you will need to copy this file to all hosts.	🟢
-h, --help	Display this help and exit.	🟢
-i, --ignore-check	Ignore the check for minimum hardware requirements.	🟢
-id, --install-dependencies	Installs automatically the necessary dependencies for the installation.	🟡
-o, --overwrite	Overwrites previously installed components. This will erase all the existing configuration and data.	🟢
-of, --offline-installation	Perform an offline installation. This option must be used with -a, -ws, -s, -wi, or -wd.	🔴
-p, --port	Specifies the Wazuh web user interface port. By default is the 443 TCP port. Recommended ports are: 8443, 8444, 8080, 8888, 9000.	🔴
-s, --start-cluster	Initialize Wazuh indexer cluster security settings.	🟢
-t, --tar <path-to-certs-tar>	Path to tar file containing certificate files. By default, the Wazuh installation assistant will search for a file named wazuh-install-files.tar in the same path as the script.	🟡
-u, --uninstall	Uninstalls all Wazuh components. This will erase all the existing configuration and data.	🟢
-v, --verbose	Shows the complete installation output.	🟢
-V, --version	Shows the version of the script and Wazuh packages.	🟢
-wd, --wazuh-dashboard <dashboard-node-name>	Install and configure Wazuh dashboard, used for distributed deployments.	🟢
-wi, --wazuh-indexer <indexer-node-name>	Install and configure Wazuh indexer, used for distributed deployments.	🟢
-ws, --wazuh-server <server-node-name>	Install and configure Wazuh manager and Filebeat, used for distributed deployments.	🟢

Questions to Discuss

Will the password-tool be maintained?

The owners of each tool will maintain the password tool. (each tool will have their password tool)

Will the cert-tool retain its current options, or will it be simplified?

Some things need to be defined about the cert-tool, but in principle, it would keep the same functions and be published as it is currently. For the moment, it would be kept in bash, and ultimately, if the changes to the assistant are successful, it would be migrated to Python. This could lead to the assistant not having the cert-tool embedded as it is currently.

Will it be possible to use a custom directory for both config.yml and the tar with the certificates, rather than requiring them to be in the same path as the wazuh-install? (Regarding the -c and -t tags)

I understand that this is a decision that we must make. Currently, this option gives the user more flexibility in handling these files. In principle, we would keep them.

Will the development mode installation still be used? (Tag -d)

As planned, no, the installation assistant should have a file with the list of URLs of the packages to be installed. In the case of development, we should use that file and replace the URLs of the packages according to our needs.

Will customizing the dashboard port still be supported? (Tag -p)

No, the installation assistant will only install the products on the default ports, customization of this will not be allowed.

Without using Filebeat for communication between the server and indexer, how will it be possible to verify that the communication between them is working correctly?

This will probably be answered later in the product itself. Wazuh server probably has some functionality to validate connectivity against indexer. We will check.

Will still be possible to do the distributed installation or only the AIO?

Distributed installation will be possible, but only with the default options. That is, only the configuration will be modified to be able to be configured in a distributed way.

Will the start-cluster option still be available?

This will depend mainly on how the Wazuh indexer is developed by OpenSearch, as it has already been announced that the security admin will be deprecated and replaced by another service. We will have to validate with our team how this will be implemented.

The security admin script will be maintained so we will have to maintain a similar option to initialize the cluster