Create logstash oss assemble process and publish it to downloads page/docker

[VijayanB]

The script to download logstash oss, install logstash-output-opensearch and re-distribute as logstash-oss-with-opensearch-output-plugin is available at https://github.com/opensearch-project/logstash-output-opensearch/tree/main/release.

We are working on adding new logstash-input-opensearch plugin. Shall we move this logstash oss assemble process to opensearch-build? Please provide feedback.

[gaiksaya]

[Triage] Hi @VijayanB , for any component building/assembling part, you can have scripts/build.sh file in your repo which takes care of everything. Example: https://github.com/opensearch-project/job-scheduler/blob/main/scripts/build.sh

If the new pluginlogstash-input-opensearch plugin is part of OpenSearch bundle it needs to be a part of the manifest and needs to have a build script like above.
Hope this answers your query!
Thanks!

[VijayanB]

@gaiksaya This is part of logstash oss, not OpenSearch. Currently, logstash-ouput-opensearch contains this assembly process ( install this plugin into logstash oss ). In future, we need something similar to what OpenSearch is doing for its plugins.

[gaiksaya]

I assume that is considered as a stand-alone component then? In that case the repository with all the codebase should contain all the build, test processes. Please see #1234, proposal section. The opensearch-build repo or code base will be Downloading the build instead of actually building the artifact.

[VijayanB]

Here repositories ( logstash-output-opensearch, logstash-input-opensearch) are plugins similar to k-NN, Alerting, AD, etc.. every plugin will have its build process to publish to rubygems.org like you rightly pointed out, but, this request is similar to how you have scripts that will generate final OpenSearch distribution with all plugins. In that case, i see two option, either create own repo with just build scripts to install all those plugins into logstash oss or move this script here

[gaiksaya]

@bbarani Can you take a look at this request?
Thanks!

[bbarani]

@VijayanB This is not on our priority list at this point in time. Having said that, our plan is to support only the signing, promoting and publishing part of any standalone artifact outside of OpenSearch distributions. We are recommending the respective teams to own the build and test process since its not sustainable to manage multiple versions along with dependency. Please take a look at our universal build design to get started on our new process. @dlvenable has worked closely with us to automate the data prepper and he can guide you with the automation here as well.

[VijayanB]

Thanks @bbarani . To be more clear, this is not building/testing set up process. This is already taken care by individual logstash plugins. In the past ( last year ), we supported only one logstash plugin, which is logstash-output-openseach, and, when we want to distribute, we needed a process which does following
Distribute as tar

1. Download logstash-oss
2. Download logstash-output-opensearch plugin from ruby gems.
3. Install downloaded plugin to logstash oss
4. Rename the logstash oss as logstash-oss-with-output-opensearch plugin
5. Upload the above artifact as tar

Distribute as Docker image

1. Create new docker file which downloads logstash oss ,
2. install output plugin
3. publish as new image

Since, it was only one plugin we hosted the above process in the output plugin repo itself. Now, we also created a new plugin called "logstash-input-opensearch". This repository will contain build, testing and publishing to rubygems as output plugin. Now, we don't want to create another logstash-oss with input plugin, instead, we should have one distribution process, which should install both input and output plugin and bundle and distribute.

IMO, it will be odd, if we host this distribution on logstash-core-oss-with-opensearch in any of the plugin, it should be created outside both the plugin. In this case, what is your recommendation? Should we create a new repository (logstash-build) just to have this distribution process or should we create inside opensearch-build a folder, which will have the distribution process ?

[VijayanB]

@dlvenable Does data prepper has similar request ?

[dlvenable]

@VijayanB , If I understand you correctly, you want to create a "Logstash OpenSearch" distribution which contains two independent projects - logstash-output-opensearch and logstash-input-opensearch.

Data Prepper is currently releasing one artifact which comes from one repository, so it is a little different. But, we are considering future changes to split some plugins out of the repository. In which case, we'd still perform our own build and then utilize the opensearch-build only for the signing and promotion process.

A reasonable solution for Logstash would probably be to create a new repository which builds both. I believe you suggested this with the logstash-build project. I might recommend that you call it logstash-opensearch or logstash-opensearch-build to make it clear that it isn't really about building Logstash per se, but creating a specific distribution.

[dblock]

A logstash-build project makes a lot of sense to me if it shares little or nothing with opensearch-build.

[bbarani]

@VijayanB I am closing this issue based on the recent comments. Feel free to re-open if you have any other questions.

[dlvenable]

@bbarani , I believe that we still want this. The idea behind this issue is to have a build process for the Logstash distribution. It does not cover automating the Gemfile creation for logstash-output-plugin.