[Meta] Create universal mechanism for releasing OpenSearch clients

[peternied]

Problem:

Today component release processes vary depending on platform and project maturity. Take OpenSearch with a large and complication release process centralizing many components and sub-components across git repositories, then consider projects like OpenSearch-py with a much smaller surface area. These are both OpenSearch projects and need to be released following consistent ways - with automation. The wrinkle in decentralization is we have requirements for how accounts, credentials, keys are managed that are locked down to a small list of people.

We need a way to create reusable centralized release process that support all of the existing and future products in OpenSearch.

Proposal 1: Enhance the OpenSearch build process via Jenkins CI

See issue history

Proposal 2: Enhance the OpenSearch release process via Jenkins CI

All components would create a manifest, much like those that exist today for OpenSearch, that tracks what is being released and how to release it. A jenkins jobfile loaded onto the CI system would know how to inspect that manifest, and take it through the stages for release. This jenkins jobfile would leverage a shared set of libraries built in the OpenSearch-Build repository that know how to pick up appropriate credentials, standard intermediate artifact locations and publishing processes.

Teams onboarding to this process would need to do the following:

1. Create a release workflow for their project that will draft a release on the GitHub repository
2. Create a jenkinsfile, setting up sections for
   • Downloading the build
   • Signing the artifacts
   • Release artifacts publishing

Handlers for Maven, Npm, PyPI, RubyGem release processes built into the existing common groovy framework and might need to be built before the component can release, but should easily reusable

---

Design tasks:

• Create a proof of concept for onboarding and detecting new releases #1539
• Could a GitHub release replace a build manifest, should we try it? #1506
• How can external workflows be authored in a way that makes sure best practices are followed? #1534
• How can OpenSearch-Build make sure it does not create a breaking change for external workflows? #1536
• What modifications are needed for the input / build / dist manifests? #1537
• Where should manifest files be placed, and what branch? #1538
• How should teams manage pre-release artifacts? #1578
• How can workflows be tested? Covered by # 1536 above

Implementation tasks:

Note; this work should not start until all design tasks have been completed as they have implications on the acceptance criteria of these tasks.

• Enable versioning for jenkins shared libraries #2489
• Create a function for a jenkins pipeline with defaults and basic configuration parameters #1606
• Inventory and create library functions for common functionality #2505
• Create a distribution representation (backed as manifest file or as a GitHub release draft) in groovy library #1608
• 2PR manual approval workflow #3111
• Documentation meta issue #2493

[peternied]

@jcgraybill @dblock @dblock @abhinavGupta16 we have talked about the release process many times, here is where I am currently thinking

[dblock]

I think we should be automating version discovery as well similarly to OpenSearch and OpenSearch Dashboards. So instead of 1. Create a manifest for their project teams onboard into a manifest CI that is capable of locating all potential versions of the component to be released and creating all of release GitHub issues from templates, manifests for the component, and Jenkinsfile.

[bbarani]

We need to make sure we not only run the integration tests rather we also add support for running BWC and perf test for the standalone components as well. We need to ensure that the standalone components we release is backwards compatible and follows semantic versioning. Also during the release of OpenSearch distribution, we need a mechanism to run automated tests against these components to make sure it works as intended against the upcoming distribution as well.

[peternied]

@bbarani Those could be great follow up enhancements as they would be supportable with this proposal.

[jcgraybill]

@peternied I really like this direction and approach. I realize that for some of the "smaller" components like opensearch-py, the common build/test process can feel like overkill today, but that's a short term concern. Build/test/deploy infra has more in common across components than different, and there are so many benefits to having consistently-applied best practices at this layer of the project.

[peternied]

Have been exploring how we should advance this design, here is what we've come up with.

Action Items

• Moved up to the top task

Design Tracks

What is the core value to OpenSearch projects?

For OpenSearch and OpenSearch Dashboards, the build / assemble into a distribution of many components is a huge win. This provides an environment to discovering build failures that are easily replicated in individual repositories. Follow on that trend, the execution of suites of tests has the same benefits of providing a bridge between all related components.

What about the value to smaller projects without several components?

For projects like Data-Prepper or OpenSearch Java Client the benefits are less clear as they already have an environment their repository, there is not much value in a second environment.

Why do products need to align with this process?

OpenSearch we centralize credentials manage/access around the @opensearch-project/engineering-effectiveness team. To use these credentials jobs need to be run on the OpenSearch-CI system. We don't recommend teams run all their build/PR/Snapshot jobs because GHA are much easier to integrate.

This makes defacto process a rarely used a release workflow to sign / publish. If build tools change GHA workflows would be updated but OpenSearch-CI workflows would also need an update, that would be easy to forget about until its time to release, causing delays when projects want to ship. It also 'locks' these projects into this build workflow.

So what would be valuable to all OpenSearch projects?

By focusing these mechanism to be release oriented, by taking in a pointer to artifacts, how to process them for release (sign with .sig files), and where to publish them (maven, artifacts.opensearch.org, npm) OpenSearch projects can leverage shared libraries for these common tasks.

This change would be best reflect in rename this process as a OpenSearch-Build model should be the universal mechanism for releasing for OpenSearch-Project

How are teams going to engage with this system?

For these smaller projects, it would be painful to commit code/configuration to other repositories. Writing workflows and manifest files into their project repository gives teams flexibility.

This flexibility creates the opportunity for unexpected results, ensuring the Jenkins workflow best practices are followed and sane defaults are used. This should be available via the use of the Jenkins shared library used for workflows.

With configuration defined in the repository and following already established patterns updates can be check for by scanning a set of registered repositories.

What would onboarding look like?

• There needs to be an relationship between the project repository and the workflow system so new manifest files can be detected and executed in the project specific workflow.
• Release processes should be identified, and if there is a release process that has not been created it needs to be added.
• The workflow should created and it should be tested to confirm it works
• The project would release once via the automation.

What about running validation?

Validate requires an environment, and if the job isn't executed daily it is going to get out of date and break. If this mechanism is focused only on releasing ready-to-go projects there are clear responsibilities.

What about teams that are interested in testing frameworks?

BWC and performance testing have been coming up as having common tracks that need resolutions. If we added features for this, it would be a better approach to decouple it from this process, that way we could leverage the best tech for the job rather than being forced into this single model.

Could a creating GitHub release be the signaling mechanism?

Interesting idea, we should look into that

[dlvenable]

@peternied , I like this concept.

What is the purpose of the manifest if the scripts themselves are in conventional locations? Does it just define which types of artifacts are produced?

Regarding project convents ("Follow common project conventions for"), I'd like to see a specific directory for build scripts. This way projects don't clutter their main directory. And it can avoid naming conflicts with build tools.

If the directory were called opensearch-build, we'd have:

• Building {repo}/opensearch-build/build.sh
• Tests {repo}/opensearch-build/integ-test.sh

[dblock]

Location of scripts is a good point, we have coded that to be scripts/, but scripts/opensearch-build/ sounds better.

[dblock]

We should re-design this and do a campaign in a release of going through all repos to adjust it accordingly.

[peternied]

FYI - I've rewritten the issue substantially as well as flushed out detail for all of the follow up action items include exit criteria

[gaiksaya]

We will be tracking publishing to new platforms if and when new requests come in. We have 2 pending ones for now which will be tracked here: #2505
We have documented the on-boarding process to 1-click release process .
Closing this issue as we have established the end to end process and also on-boarded most of the components existing today.