DTT1 - Iteration 1 - Design solution

[rauldpm]

EPIC: #4495

Description

It is necessary to study and analyze the design to follow for the implementation of the issue #4495, so that the established requirements are taken into account and an initial development direction can be established.

This first phase will consist of multiple brainstorms so that the proposals can be studied and evaluated, defining benefits and cons.

It is necessary to take into account the entire possible spectrum, from where the code will be stored, the creation or not of a new Jenkins to the final part of the process in which the results are reported and handled.

This issue will be worked on in a shared Google Drive Document: Link

[rauldpm]

Update report

An initial brainstorming has been carried out with @jnasselle from which the following questions have been raised:

1. Where will the new code be stored? Should a new repository be created?
2. Will the current Jenkins be used or will a new Jenkins be dedicated?
3. What tools are we going to use, Python, Ansible, Groovy, Docker? Proposals
4. What tool are we going to use for the tasks? Are tasks going to be defined using Ansible? Are we going to use Python for those tasks too?
5. How do we perform tests that check logs from different nodes? Do we want to make it centralized? Do we want to divide the execution of the tests sequentially? Are we going to use a monitor node from where all the execution will be centralized
6. How are we going to modularize the execution of the pipelines?
7. What does the execution of each modular pipeline receive and return?
8. What type of resources are we going to use? EC2, ECS, Vagrant?

---

• Testing semantics
  • Analyze each test and figure out could be executed as-it in the host under test
  • Check test and (order) dependencies between them. Early-abortion should be implemented
• Reliability
  • How can guarantee that the results reflect the true nature of the product status?
• Lightweightness,
  • This concept can have multiple meanings. We can relate this to simplicity
• Speed
  • Allow as much paralelization as posible. Horizontal scalability
  • Asynchronous /event handling rather than timeout
  • Resources caching

[rauldpm]

Update report

• Several meetings have been held to discuss the points set forth in the previous commentary with @jnasselle and @fcaffieri.
• It has been determined that the alternative tool to Blue Ocean should allow the visualization of the pipeline execution and nothing else.
• The definition and use of the tools to be used has been expanded.
• The roadmap for the execution of the tasks and the subsequent tests has been defined.
• The structure of the modular pipelines has been re-evaluated, and the parameters and relationship of each pipeline have been defined.
  • The parameters will be minimal (branch and YAML).
  • The YAML will determine the infrastructure and will be used as a relationship between the pipelines so that everything necessary can be identified.
  • It has been decided to use Groovy for the Jenkinsfiles and Python will be the language used for the rest, including the classes and tests, following the good practices of Jenkins, also, this way, the local usability of the processes is allowed.
  • We have investigated and tested the possibility of generating the content of a pipeline parameter dynamically based on the content of a GitHub file (referenced by a branch parameter). It is not possible as it is a private repository.
• The order of the tests has been reevaluated since a problem has been detected in the install/upgrade sequence: Deployability testing tier 1 #4495 (comment)

[rauldpm]

• I have to stop working on this issue to work on an issue stopper for 4.6.0 Review changes in the Wazuh indexer upgrade wazuh-packages#2460 and Review changes in the Wazuh indexer upgrade wazuh-packages#2460
• Working on DTT1 - Design and develop PoC #4524

[fcaffieri]

Conclusions

The objective of this issue was to generate a basic and modular architecture, to begin to see the problems that may arise with the modular implementation of DTT1. This objective was completed in the next branch and after PoC iteration 1.
A modular design was defined with the following architecture and it was versioned and implemented at https://github.com/wazuh/wazuh-qa/tree/enhancement/4495-deployability-tier-1

Architecture:

---

---

---

---

Problems found:

• Difficult to carry out modularization due to intrinsic dependencies between modules, due to the nature of the tests.
• Collector module is not necessary, it was absorbed by the Observability module.
• An improvement is required on all modules, so that they:
  • Perform schema validation with pydantic. To validate the inputs they receive.
  • Be self-sufficient and independent, they can be called from any point without needing to receive too many parameters.
  • Make diagrams of each one with a certain level of detail, which allows the understanding of each one.
  • Redefine the inputs and outputs of each one, since it was not finalized in the PoC.
• Investigate the need to implement a flow orchestrator, in order to be able to easily define the use cases at a high level, so that it can then execute each of the modules depending on the case.