Update package generation tools

[AlexRuiz7]

Description

In the context of the packages redesign project, new requirements have been defined or have changed slightly, so we need to update our tools to accomplish with these requirements, which in our case are:

• Distinction between development and production packages. The GH workflow must provide a way to switch between the generation of these 2 packages types (is_stage input).
• Production packages follow a different naming convention, which corresponds to current packages naming (see wazuh-indexer packages list). Production packages do not contain the commit's SHA.
• By default, all packages use 0 as revision, unless stated otherwise. This means that the GH Workflow should allow providing the revision number as input.
• The GH Workflow must auto skip if there is already a package in the S3 bucket from the same commit, unless the package is of type stage. In such case, the old package must be overwritten with the new one.

For a complete list of requirements, refer to https://github.com/wazuh/internal-devel-requests/issues/187.

[AlexRuiz7]

The curent tooling to generate packages were not designed with the capapbility of sypporting different naming conventions in mind. Not only that, but also the packages' metadata must be different.

We'll then distinguish 2 types of packages:

• pre-release or development packages
• release packages

Depending on the package type, the release packages follow this naming convention:

deb ~~ <SubsystemName>_<VersionNumber>-<Revision>_<Architecture>.<PackageType>
rpm ~~ <SubsystemName>_<VersionNumber>-<Revision>.<Architecture>.<PackageType>

====

deb ~~ wazuh-indexer_<VersionNumber>-<Revision>_amd64.deb
rpm ~~ wazuh-indexer_<VersionNumber>-<Revision>.x86_64.rpm

====

wazuh-indexer_4.7.3-1_amd64.deb
wazuh-indexer-4.7.3-1.x86_64.rpm

As the GH workflow and tooling currently in the default branch (4.9.0) does already accomplish with the requirements for development packages, we'll replicate that for release packages, forking the workflow in 2 pipelines.

flowchart LR
    A[Build] -->|inputs| B(name generator)
    B --> C{is_production}
    C -->|Name One| D[pre-release]
    C -->|Name Two| E[release]

Loading

In order to do that, the current workflow becomes the pre-release pipelined, so needs to be converted to a reusable workflow, and must be able to communicate the package name, build parameters and the packages themselve between the different levels of reusable workflows.

The build workflow in the pipeline will take the controller role, reading and evaluting the inputs, deciding which workflow to run and interfacing with them (data sharing). This workflow is of new creation.

The pre-release pipeline will be different from the release workflow in what it doesn't need to check if the package already exists in the S3 bucket, as thes packages will always be uploaded.

Both pre-release and release packages will receive the name of the package to build from the controller.

It is very possible that we need to refactor the bash scripts used to manage the packaging process (see #180) to completely fullfill the requirements.

[AlexRuiz7]

Working on https://github.com/wazuh/internal-devel-requests/issues/187#issuecomment-2018579969

Custom metadata

@/wazuh/devel-indexer and @/wazuh/devel-dashboard teams must validate:
Alphanumeric revisions can be used in package generation

[AlexRuiz7]

The combination of packages to be generated are specified to the workflow using arrays. These arrays are then used to generate a dynamic build matrix in GitHub Actions.

      distribution:
        description: '[ "tar", "rpm", "deb" ]'
        type: string
        default: '[ "rpm", "deb" ]'
      architecture:
        description: '[ "x64", "arm64" ]'
        type: string
        default: '[ "x64" ]'

References: https://www.kenmuse.com/blog/dynamic-build-matrices-in-github-actions/

[AlexRuiz7]

The size of the packages has increased quite a bit compared with the previous version (4.7.3).

wazuh-indexer_4.7.3.x86_64.rpm	673 MB
wazuh-indexer_4.9.0.x86_64.rpm	800 MB
opensearch_2.12.0.x86_64.rpm	782 MB

From these sizes, we understand that OpenSearch has grown up, so it's expected that our packages grow too. There is only an 18 MB difference, which can correspond to some additional files we include into the packages, such as bash scripts or the alerts index template.

[AlexRuiz7]

Custom metadata checks

RPM

bash-5.2# rpm -qi /packages/wazuh-indexer-4.9.0-1.x86_64.rpm 
Name        : wazuh-indexer
Version     : 4.9.0
Release     : customrev7
Architecture: x86_64
Install Date: (not installed)
Group       : Application/Internet
Size        : 1066800744
License     : Apache-2.0
Signature   : (none)
Source RPM  : wazuh-indexer-4.9.0-customrev7.src.rpm
Build Date  : Tue Mar 26 12:34:23 2024
Build Host  : fv-az659-298.luykedtozraezimdvwvtsy4hga.cx.internal.cloudapp.net
Packager    : Wazuh, Inc <info@wazuh.com>
Vendor      : Wazuh, Inc <info@wazuh.com>
URL         : https://www.wazuh.com/
Summary     : An open source distributed and RESTful search engine
Description :
Wazuh indexer is a near real-time full-text search and analytics engine that
gathers security-related data into one platform. This Wazuh central component
indexes and stores alerts generated by the Wazuh server. Wazuh indexer can be
configured as a single-node or multi-node cluster, providing scalability and
high availability.
For more information, see: https://www.wazuh.com/

DEB

root@166d9bf9c8d0:/# dpkg -I /packages/wazuh-indexer_4.9.0-1_amd64.deb 
 new Debian package, version 2.0.
 size 837113394 bytes: control archive=27479 bytes.
    1704 bytes,    30 lines      conffiles            
     676 bytes,    13 lines      control              
  103182 bytes,   994 lines      md5sums              
    1470 bytes,    53 lines   *  postinst             #!/bin/bash
    1118 bytes,    31 lines   *  preinst              #!/bin/bash
     828 bytes,    26 lines   *  prerm                #!/bin/bash
 Package: wazuh-indexer
 Version: 4.9.0
 Architecture: amd64
 Maintainer: Wazuh Indexer Team <info@wazuh.com>
 Installed-Size: 1036878
 Section: web
 Priority: optional
 Homepage: https://www.wazuh.com/
 Description: Wazuh indexer is a near real-time full-text search and analytics engine that gathers security-related data into one platform.
  This Wazuh central component indexes and stores alerts generated by the Wazuh server.
  Wazuh indexer can be configured as a single-node or multi-node cluster, providing scalability and high availability.
  Documentation can be found at https://documentation.wazuh.com/current/getting-started/components/wazuh-indexer.html
 License: Apache-2.0

[AlexRuiz7]

A convenience workflow (build_single.yml) has been created to generate one package per call, so QA can normalize the invocation of the workflows across the repositories. The workflow calls the build.yml workflow with the agreed parameters.

[AlexRuiz7]