[RFC] Improving public engagement for OpenSearch release process

[bbarani]

Summary:

OpenSearch is a fast-growing community-contributed project and currently has around 600+ contributors who actively participate in the growth of the project day-to-day. OpenSearch has been piloting the process of hosting public meetings for some time, where public members can join the meetings and actively participate synchronously. As an open-source project, we aim to establish shared ownership of OpenSearch with the public through an open governance model. We're already publishing OpenSearch release updates via GitHub and Slack. Our next goal is to adopt an open release model, allowing public participation in OpenSearch releases. This proposal seeks feedback on our current release process, roles, and mechanisms to improve public involvement.

Motivation:

OpenSearch currently release major, minor and patch versions on a regular cadence as listed on this release page. The infrastructure to build, test and release artifacts are available at https://build.ci.opensearch.org/. A public release GitHub issue using this release template is created in the opensearch-build repo along with individual component GitHub issues created using this template for all plugins participating in the release. A primary release manager along with secondary release managers corresponding to participating repos are assigned for a release. The primary release manager goes over the individual release issues periodically across the repos and engage the secondary release managers to take appropriate action as needed throughout the release process. A release call is scheduled with necessary stakeholders to complete the release process.

The status of a release is currently tracked in public GitHub issue, but we believe there is additional opportunity for public collaboration during releases. We strongly believe that involving anyone who wants to be involved in the public is an important step an important step to improve the trust and transparency within the public along with getting additional support, feedback from external contributors for closing the gaps, and improvements to the overall release process.

This scope of this RFC doesn’t include standalone OpenSearch client releases that has been already fully automated and self-serviceable by anyone as described in this GitHub issue.

It would be helpful to go over the current roles and responsibilities before diving deep in to the release process.

Roles and responsibilities:

The below roles and responsibilities are currently involved in OpenSearch release cycle process, we want to collaborate with the public to improve it further by adding, modifying roles and responsibilities (as needed).

Role	Responsibilities
Release manager	Responsible for planning, testing, tracking, release, deployment, communication, and risk management. Collaborate with technical and leadership team to finalize a release schedule and plan for OpenSearch Responsible for managing the release lifecycle process that includes, * Scheduling the release * Co-ordinating between teams * Broadcasting release candidate information to gather votes * Execute automated tests * Signing the artifacts * Deploying release artifacts as per the schedule * Completing post release activities Maintain proper coordination between multiple participating teams to update the project related information in publicly accessible platforms
Repository owner	Responsible for assigning a repo level release manager / POC for a specific release Work closely with release manager to identify, remediate possible gaps for a release corresponding to a specific feature on this repo Ensure sanity tests are executed and documented by assigned release manager Help remediate issues found during testing Surface any gaps to release manager in timely manner Provide votes for finalizing release candidate
Maintainers	Perform sanity test on the release candidate Surface any issues, gaps to release manager in timely manner Help remediate issues found during testing Provide votes for finalizing release candidate
Leadership team	The leadership team is responsible for, High-level technical direction of the OpenSearch product The roadmap of the OpenSearch project and releases Creating appropriate Working Groups ( User Feedback, Documentation etc..) to gather the necessary public feedback before making decisions Technical resources (e.g., code repositories, servers) Maintaining maintainers, committers list Call for closing vote / vote to table any issue Go / No-Go vote on product releases

OpenSearch / OpenSearch dashboards Release:

At a high level. OpenSearch release process involves collaboration with multiple maintainers from OpenSearch, OpenSearch dashboards, website, documentation GitHub repositories that are participating in a release. OpenSearch follows de-centralized development model where the development for core engine, plugins happens across different geographical location. The distribution build (OpenSearch, OpenSearch Dashboards) integrates the code across all these repositories for all supported versions via automated CI / CD process to generate, test and publish nightly builds.

These are the pre-requisites that needs to be closed out before the start of release cycle process.

Pre-requisites:

1. Release manager for a specific OpenSearch release is finalized and assigned 28 days before a planned release.
1. Once the release manager is finalized, his details is assigned to the release GitHub issue created in opensearch-build GitHub repository and also broadcasted to public slack channel.
2. Release issue corresponding to major, minor, patch is created using the release template.
1. Release issue is immediately after the OpenSearch core bumps the version to next major, minor, patch version.
3. Release manager is responsible for creating child component issues on all Github repositories participating in a release.
4. Release manager is responsible for releasing both OpenSearch and OpenSearch dashboards.
1. This is subject to change in future as OpenSearch and OpenSearch dashboards are technically independent products with their own release cycle process.
5. Release manager will obtain all required access during release cycle process..
1. This step needs to be completed 5 days before the scheduled preparation phase for a release. Release manager works with the members of opensearch-build team to get all necessary access. They can also reach out to the members via OpenSearch public slack channel (#release)
6. Release manager will co-ordinate with all participating GitHub repo maintainers via OpenSearch public slack channel (#release)
7. Any questions, issues, feedback related to the release cycle, process, mechanism is discussed in OpenSearch public slack channel (#release)

Current release cycle process:

Note: The PR that describes the end to end OpenSearch release process is in review. The information links added to each of the following steps will be part of the opensearch-build GitHub repository once the PR is merged.

• Preparation (14 days from release date) - Owner [Repository owner, leadership team, Release manager ]
• This phase involves preparing for a release that includes assigning a release owner, updating the release cycle dates, finalizing the feature for a release as listed on the roadmap along with creating component release issues across all participating repos in a release.
• Release branch readiness (Not applicable for patch release) (10 days from release date) - Owner [ Repository owner, Release manager ]
• This phase requires that every team participating in a release have their release (candidate) branch created by this date. The distribution build workflow will also start using the release branch to create release candidate instead of .x branches.
• Code Complete / Feature freeze (7 days from release date) [Release manager, Repository owner]
• All maintainers of OpenSearch GitHub repositories targeting a specific release merge their PR’s ( i.e.e vMajor.minor.patch labeled PRs) to release branch branch before release manager creates the release candidate.
• Release candidate creation (6 days from release date) [Release manager]
• The automated build workflows will create release candidates with the finalized commit and publish the docker images and associated distribution artifacts for testing. The release manager will broadcast the release candidate to public for sanity tests.
• Release testing (5 - 2 days from release date) [Release manager, Maintainers, Repository owners]
• Automated integration, BWC and performance testing is kicked off against the release candidate. The release candidate information is also broadcasted to wider group for another round of smoke testing using the standard communication template.
• Go / No-Go voting (1 day from release date) [Release manager, leadership team, Repository owners]
• After all validations and verification, promote the artifacts to production bucket for the final release after getting the sign off from leadership team committee through Go / No-Go voting process.
• Release (Release day!) [Release manager, leadership team, Repository owners ]
• Release the artifacts to production distribution channels, work with website team to update the website and inform the public of the release.
• Post release activities (Release day!) [Release manager]
• This steps ensures that the tags are created for all participating repos, incrementing versions for next patch, minor release along with house cleaning activities. Release manager also collaborates with the website and documentation repo owners to publish the website updates, blog posts and documentation update corresponding to a release.

Note: We have put out a proposal to move away from release train model with defined release date to release window model with a defined release candidate (RC) creation date. We will iterate on “Release candidate” creation along with “Release testing” phase until we finalize the release candidate. Release day will be 2 days after then final RC creation date. Leadership team can also recommend to skip a release version if we are not able to finalize the release candidate within a reasonable time period to avoid cascading effect to releases.

---

Frequently asked questions (FAQ):

How are the release managers finalized for a release?

Release managers are finalized based on volunteer model. A request for release managers for a specific release will be broadcasted in GitHub as we well as OpenSearch Slack channel and the release manager will be selected based on the first come first serve basis.

Note: Release managers has to be a maintainers of Github repositories under OpenSearch project.

Where can I see the finalized features for a specific release ?

You can visit OpenSearch public roadmap to get the comprehensive list of features scheduled to be released in specific OpenSearch version.

Who are the members of the leadership team?

The leadership team is currently seeded by the below list of AWS employees.

• Carl Meadows
• Anandhi Bumstead
• Eli Fisher
• Charlotte Henkle
• Xin Liu
• Nick Nize
• Daniel (dB.) Doubrovkine
• Barani Bikshandi

This is the current list but feel free to provide your feedback to add additional members to the team via OpenSearch public slack channel

Does OpenSearch and OpenSearch dashboards have separate release timeline ?

Currently OpenSearch and OpenSearch dashboards are released at same time (with same version number) as part of OpenSearch release.

How can I participate in release process?

If you are not part of the listed roles, you can still actively participate in release process by sanity testing the release candidate along with providing your feedback, votes for finalizing the release candidate.

How can I become a maintainer of a repository?

Any new members who are interested in becoming maintainers on specific Github repository need to be nominated by the current maintainers through voting process as described here.

---

Next steps:

All the steps listed above in release cycle process are already accessible by the public and the release steps, process are documented in public Github issue. One of the primary focus of this RFC is to surface the release process, roles and responsibilities along with detailed release guide to improve public engagement in OpenSearch release process.

We would like your feedback on,

• Improving the public engagement for OpenSearch release process
• We would like a broader participation in OpenSearch release cycle process. We have updated the release template, SOP and collaboration channels to make the OpenSearch release process more accessible by the public. We need your feedback on any gaps, process technical limitations that would help improve the engagement.
• Formulating process, mechanism to add new roles to release process
• Please provide us feedback to draft a proposal to add, modify roles to OpenSearch release process
• Improving, simplifying the steps involved in current release cycle process
• Help us improve the OpenSearch release process by providing your inputs. We welcome both technical and process inputs.
• Improving the quality of OpenSearch artifacts
• Provide us inputs on improving the quality bar of every OpenSearch releases.
• General comments around the current process including anything that's confusing
• Feel free to provide any other ad-hoc inputs related to release process, public engagement, roles and responsibilities.

We would like get your feedback and suggestions on this RFC to make the release process as seamless as possible. We look forward to collaborating with the public.

relates #166

[acarbonetto]

Thanks @bbarani for this document! Having everyone aligned on a release process will be extremely helpful. A couple of questions/requests:

1. Could we create a separate release channel for each new release. For example: #release_2.8.0, #release_2.9.0, #release_2.8.1, etc... so that we stay on-topic and don't have any confusion. We can archive the channel once release retrospective is completed for all parties.
2. Can we start the release process for dependency libraries earlier, while plugins and repos without dependencies start later? common-utils, for example: we could get release ready earlier so that all the plugins (such as ml-commons) depending on common-utils are unblocked.
3. Report breaking API and library changes to the release channel to solicit feedback and notify dependent plugin owners of the change(s).
4. Advertise the changes and comments from release issues (see [RELEASE] Release version 2.8.0 opensearch-build#3434) into the release channel for slack notifications.

[peternied]

Thanks for documenting all of this the clarity is a great improvement - could you transform this issue's description into a pull request on the RELEASING.md so this is codified and committed to the repository?

[nknize]

Thanks for documenting all of this the clarity is a great improvement - could you transform this issue's description into a pull request on the RELEASING.md so this is codified and committed to the repository?

+1.. let's get this into an actual reviewable PR so we can comment inline.

Can we add some concrete language to some of these steps so folks know what goes into each one?

For example:

1. Release manager for a specific OpenSearch release is finalized and assigned 28 days before a planned release.
   • What's the process here? Do we request volunteers on slack #releases channel?
2. Once the release manager is finalized, his details will be assigned to the release GitHub issue created in opensearch-build GitHub repository and also broadcasted to public slack channel.
   • Who does this? The release manager? Assigns themselves as the assignee in github? Posts a slack message?
3. Release manager is responsible for creating child component issues on all Github repositories participating in a release.
   • Can we automate the process if it isn't already? If it is automated, is there a doc the RM can use to show them how to do this?

Let's start with that here and then have one single FOOLPROOF page an RM can follow even if they've never come close to the release process.

[prudhvigodithi]

2. Can we start the release process for dependency libraries earlier, while plugins and repos without dependencies start later? common-utils, for example: we could get release ready earlier so that all the plugins (such as ml-commons) depending on common-utils are unblocked.

Hey @acarbonetto, it should be possible to onboard plugins earlier provided the version increment for core and common-utils is completed and added to the input manifest, this depends on how quick/early an RM and the plugin team coordinate to achieve this. We can also think of an automation where the plugins that does not have dependencies are force incremented along with core (should be ok since we are releasing all the plugins along with core as a single bundle, rather than an isolated release for each plugin).

[prudhvigodithi]

Release manager is responsible for creating child component issues on all Github repositories participating in a release.

• Can we automate the process if it isn't already? If it is automated, is there a doc the RM can use to show them how to do this?

Even though this is not required for patch release, +1 for the automation where the component release issues are created along with the main release issue part of the build repo.

[reta]

@bbarani thanks a lot for the proposal, I think this is a HUGE step in right direction, I would like to bring a few notes / questions:

• at the moment, all related release builds are happening on Jenkins (https://build.ci.opensearch.org/), this is a publicly available instance (:clap: ) but I suspect release manager might need to rerun some builds on demand, either because of hiccups or new fixes, this is not possible at the moment for non-AWS collaborators
• at the moment, the release tracking includes the components under OpenSearch project only (please correct me if I am wrong), but I suspect the community ( plugins, and in future, extensions authors) may also be interested in including their project into the release (fe, to increase the community awareness), would that make sense?

[bbarani]

Thanks for your feedback @reta . Please find my comments inline.

@bbarani thanks a lot for the proposal, I think this is a HUGE step in right direction, I would like to bring a few notes / questions:

• at the moment, all related release builds are happening on Jenkins (https://build.ci.opensearch.org/), this is a publicly available instance (👏 ) but I suspect release manager might need to rerun some builds on demand, either because of hiccups or new fixes, this is not possible at the moment for non-AWS collaborators

Currently only specific group of folks under AWS have access to execute, modify builds but we will be expanding the access to all (including external members) release managers who are interested in participating in OpenSearch releases as well.

• at the moment, the release tracking includes the components under OpenSearch project only (please correct me if I am wrong), but I suspect the community ( plugins, and in future, extensions authors) may also be interested in including their project into the release (fe, to increase the community awareness), would that make sense?

We currently don't include plugins outside of OpenSearch GitHub organization in to distribution bundle. We don't have a process defined to include plugins outside of https://github.com/opensearch-project/ with clear on-boarding process, security review process along with SLA, roles and responsibilities yet.

@reta We welcome your inputs to improve the process there. Feel free to provide any feedback, inputs to draft a process to close this gap.

Having said that, the release manifests dictates the plugin list in a release bundle.

[bbarani]

Thanks for documenting all of this the clarity is a great improvement - could you transform this issue's description into a pull request on the RELEASING.md so this is codified and committed to the repository?

Sure, we will add the summary, motivation along with roles and responsibilities to this RELEASE_PROCESS_OPENSEARCH.md soon. CC: @prudhvigodithi

[bbarani]

Thanks for documenting all of this the clarity is a great improvement - could you transform this issue's description into a pull request on the RELEASING.md so this is codified and committed to the repository?

+1.. let's get this into an actual reviewable PR so we can comment inline.

Can we add some concrete language to some of these steps so folks know what goes into each one?

For example:

1. Release manager for a specific OpenSearch release is finalized and assigned 28 days before a planned release.

   • What's the process here? Do we request volunteers on slack #releases channel?

Yes, the plan is to request for release managers for upcoming release immediately after the current release as part of post release activities. We will update the post release activities section in RELEASE_PROCESS_OPENSEARCH.md with these details.

2. Once the release manager is finalized, his details will be assigned to the release GitHub issue created in opensearch-build GitHub repository and also broadcasted to public slack channel.

   • Who does this? The release manager? Assigns themselves as the assignee in github? Posts a slack message?

Release manager might not have access to the OpenSearch build repo to self-assign the GitHub release issue but members of @opensearch-project/engineering-effectiveness will help with assigning them to release issue.

3. Release manager is responsible for creating child component issues on all Github repositories participating in a release.

   • Can we automate the process if it isn't already? If it is automated, is there a doc the RM can use to show them how to do this?

@prudhvigodithi is looking in to automating this process so that the child issues are automatically created on all plugin repos when parent release issue is created.

Let's start with that here and then have one single FOOLPROOF page an RM can follow even if they've never come close to the release process.

[prudhvigodithi]

Release manager might not have access to the OpenSearch build repo to self-assign the GitHub release issue but members of @opensearch-project/engineering-effectiveness will help with assigning them to release issue.

I have added a new section Upcoming Release Preparation to handle this.

[prudhvigodithi]

Sure, we will add the summary, motivation along with roles and responsibilities to this RELEASE_PROCESS_OPENSEARCH.md soon. CC: @prudhvigodithi

@bbarani should this be part of the https://github.com/opensearch-project/.github/blob/main/RELEASING.md file as the file RELEASE_PROCESS_OPENSEARCH.md can focus on just the release steps.

[bbarani]

Sure, we will add the summary, motivation along with roles and responsibilities to this RELEASE_PROCESS_OPENSEARCH.md soon. CC: @prudhvigodithi

@bbarani should this be part of the https://github.com/opensearch-project/.github/blob/main/RELEASING.md file as the file RELEASE_PROCESS_OPENSEARCH.md can focus on just the release steps.

I think it makes sense to combine both motivation and summary in to one section and also add roles / responsibilities to the same doc.

[prudhvigodithi]

Release manager is responsible for creating child component issues on all Github repositories participating in a release.

• Can we automate the process if it isn't already? If it is automated, is there a doc the RM can use to show them how to do this?

This process is automated now, along with the global release issue created in the build repo (Example for 3.0.0), the individual component (plugin repos) release issues are also auto-created and links back to the global release issue part of the build repo (Example for 3.0.0 component release issues).

[gaiksaya]

Hello everyone,
I created an issue that describes the current process, the gaps as well as an approach to make the release process more efficient and hands-free.
Please take a look and provide feedback.
Thanks!
opensearch-project/opensearch-build#5171