Building an OpenSearch and OpenSearch Dashboards Distribution

The distribution workflow builds a complete OpenSearch and OpenSearch Dashboards distribution from source. This system performs a top-down build of all components required for a specific OpenSearch and OpenSearch Dashboards release, then assembles a distribution. The input to the system is a manifest that defines the order in which components should be built. All manifests for our current releases are placed in manifests folder. For older versions, please see legacy-manifests folder.

What are manifests?

Before we start to build, let's understand what is manifest and the types of manifest that you will come across. Manifests are the single source of truth for building and testing the distributions. There are 3 types of manifest, each with a difference schema.

• Input Manifest: As the name suggests, input manifest is the starting point and source of building OpenSearch and OpenSearch Dashboard distributions. See the schema of the input manifest. This manifest consist of CI docker image that contains all the dependencies to build an OpenSearch and OpenSearch Dashboard distribution. It also contains the components' GitHub repository URLs, reference to build, basic CI checks as well as platforms the components supports.

• Build Manifest: Build Manifest is the interim manifest that is generated after components are built. See the schema for build manifest. You can find those manifests in builds folder locally. This manifest contains interim artifacts such as zips, libraries, maven artifacts, etc. See sample build manifest.

• Distribution Manifest: Distribution manifest, as the name suggests is the final manifest bundles with the distribution. See the schema for the distribution manifest. This manifest contains all the information about a distribution and the components bundled in that distribution.

Building from Source

Input manifests are the source of building the distributions. Each input manifest also contains the docker image that can be used to build these distributions and avoid installing bunch of packages. See CI image details in the manifest.

To run locally:

 docker run -it -d --entrypoint bash opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2 -e JAVA_HOME=/opt/java/openjdk-20

Then build from source:

./build.sh manifests/3.0.0/opensearch-3.0.0.yml 

This builds OpenSearch 3.0.0 from source, placing the output into ./builds/opensearch.

See build workflow for more information.

Assembling a Distribution

Assembling a distribution needs build manifest as an input.

./assemble.sh builds/opensearch/manifest.yml

The assembling step takes output from the build step, installs plugins, and assembles a full distribition into the dist folder.

See assemble workflow for more information.

Building Patches

A patch release contains output from previous versions mixed with new source code. Manifests can mix such references. See opensearch-1.3.1.yml for an example.

OpenSearch is often released with changes in opensearch-min, and no changes to plugins other than a version bump. This can be performed by a solo Engineer following a cookbook. See also opensearch-build#1375 which aims to automate incrementing versions for the next development iteration.

OpenSearch-Min Snapshots

Snapshots for OpenSearch core/min can be downloaded and used in CI's, local development, etc using below links:

Linux:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-linux-x64-latest.tar.gz

Macos:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-darwin-x64-latest.tar.gz

Windows:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-windows-x64-latest.zip

CI/CD Environment

We build, assemble, and test our artifacts on docker containers. We provide docker files in docker/ci folder, and images on staging docker hub repositories. All Jenkins pipelines can be found in jenkins. Jenkins itself is in the process of being made public and its CDK open-sourced.

See jenkins and docker for more information.

Build Numbers

The distribution url and the build output manifest include a Jenkins auto-incremented build number. For example, the manifest from OpenSearch build 5905 contains the following.

build:
  name: OpenSearch
  version: 2.2.0
  platform: linux
  architecture: x64
  distribution: rpm
  id: '5905'

Latest Distribution URL

Use the latest keyword in the URL to obtain the latest build for a given version. For example https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/rpm/dist/opensearch/manifest.yml redirects to build 5905 at the time of writing this.

The latest keyword is resolved to a specific build number by checking an index.json file. This file has contents such as this.

{"latest":"5905"}

The file is updated when a distribution build job is completed for the given product and version (or is created when such distribution job succeeds for the first time). Since one distribution build job consists of multiple stages for different combinations of distribution type, platform and architecture, the index.json is only modified once all stages succeed. With this said, the latest url only works when the distribution build job succeeds at least once for the given product and version.

The resolution logic is implemented in the CloudFront url rewriter. The TTL (time to live) is set to 5 mins which means that the latest url may need up to 5 mins to get new contents after index.json is updated.

All the artifacts accessible through the regular distribution url can be accessed by the latest url. This includes both OpenSearch Core, OpenSearch Dashboards Core and their plugins.

For example, you can download the latest .tar.gz distribution build of OpenSearch 2.2.0 directly at https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.2.0-linux-x64.tar.gz, without having to first download and parse the complete build manifest.

For plugin artifacts, you can also use the latest keyword to get the latest plugin artifacts with a known version. E.g. in order to get performance-analyzer x64 tarball artifacts for 2.1.0, you can obtain it with link https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip, which will redirect you to https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip.

For bundled artifacts, here are some examples for LINUX and Windows:

• Linux Tar: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz
• Windows Zip: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/windows/x64/zip/dist/opensearch/opensearch-2.4.0-windows-x64.zip