[Meta] OpenSearch Release Improvements

[prudhvigodithi]

Coming from #3676 A Central release workflow to support OpenSearch public release and #2080, following are the issues that can help improve the OpenSearch Release process. Once the following issues are achieved it would be easy to achieve the long term goal of one-click release or having an automated release tool that can release the OpenSearch bundle.

Please consider the following suggestions as areas for improvement in the OpenSearch release, aiming to enhance the efficiency and reduce the time and effort required for a release manager.

For more details about the OpenSearch release process, refer the release process guide RELEASE_PROCESS_OPENSEARCH.md

Enhancements

• Circuit Breaker for AUTOCUT issues
  There are lot of AUTOCUT issues created and closed because of flaky tests, have a mechanism to reduce the number of AUTOCUT issues created.
  Related Issue: [Feature request] Detect flaky distribuiton build failures and integration test failures #4171

• Central workflow to execute pre-release promotion jobs.
  Related issue: Central workflow to execute pre-release promotion jobs. #4136
  To have one single workflow to execute all the pre-release and all promotion jobs.

• Proper Labelling for AUTOCUT issues.
  Have proper labels added to the created AUTOCUT issues, example like if the failure is per distribution add the distribution label to the created issues. Later it would be easy to filter the issues per label.

• Consolidated Release notes automation
  The process to gather the consolidated release notes and raise a PR is now done manually by a release manager, need an automation to generate and raise a PR.
  Related Issue: Automate consolidated release notes creation #4009

• Jenkins upgrade Cycle.
  Upgrade the Jenkins, come up with a quarterly upgrade path based on released LTS version
  Related Issue: Jenkins Upgrade Cycle opensearch-ci#333

• Split Public Jenkins into multiple masters.
  Related Issue: Jenkins Multi Master support opensearch-ci#373

• Incremental builds
  Currently we are building all components sequentially and we have to rebuild all components to accommodate even small commit changes. This incremental build process will allow to rebuild only specific updated artifacts and save the unnecessary time on rebuilding other unchanged plugins. It call reduce the distribution bundle generating time from 1 hour to 10 minitues ideally.
  Related Issue: Introducing incremental builds #3922

• Refactor usage of existing index.json used in the latest URL.
  We plan to store the latest of each distribution type separately in different index.json files and retrieve the latest from the corresponding index.json that is applicable based on the distribution type specified in the url.
  Related issue: Refactor the usage of index.json file for both OS and OSD. #4151

• OpenSearch Build repo versioning framework (Resolves the need of branching).
  Related Issue: [POC] Build repo versioning framework. #2490

• OpenSearch Playground with daily build artifact.
  Related Issue: Create a "nightly" instance of Playground that gets updated everyday with the daily build. opensearch-devops#129

• Integrate Validation mechanism with every OS and OSD build.
  Related Issue: Integrate Validation mechanism with OS and OSD builds #3441

• Mechanism to auto create the release branch for the plugin repos.
  Related Issue: Auto create release version branch from .x branch on start of release cycle process #4005

• Introduce placeholder in manifests to allow skip the component or use the placeholder component for any other use-case.
  Related Issue: Add placeholder plugins when auto-generating next version's manifests #3707

• Open source the signing mechanism
  Related Issues: Open-source signing infrastructure #118,
  [Renew Key] Upcoming expiration of our current sub public key (expire on 20240512) #3468

• Once the RC is finalized, we run several tests on demand, having a central workflow that takes input of the RC numbers, runs all the test jobs sequentially and post the results in different mediums. This should include the benchmark tests as well.

• One central workflow to trigger all promotion job for releasing. Have a fall back mechanisms in place.

• Option to choose types of promotion in case of any job failure. To is to make sure the promotion jobs are only incremental from failure.

• CI docker images for build and test should be maintained per version. [Bug]: CI docker images for build and test should be maintained per version #4233

• Add Cloudfront Invalidations
  Add Cloudfront cache invalidation after each promotion #4262

• Add rollback after artifacts promoted to the prod bucket
  Add rollback after artifacts promoted to the prod bucket #4263

Bugs

• Merge the Jenkins Create/Close Github libraries to central library
  Today we have multiple libraries for creating and closing the github issue which is adding some confusion and more effort to manage the libraries. Need to merge the logic into one library and use it for all Github actions.
  closeBuildSuccessGithubIssue closeGithubIssue createBuildFailureGithubIssue createGithubIssue

• S3 Upload fails randomly
  Related Issues:
  [BUG] Transient extension mismatch in the distribution upload step #3097
  [BUG] s3upload weirdly switch architecture value during upload sha512 for snapshots opensearch-build-libraries#319

• New ci-runner user cause weird failures in integ-tests
  Related Issues:
  [BUG] Integration tests for dashboards failing with the image/latest we are currently using #4241

• Version Increment bug with OSD.
  Related Issue: [Bug]: Unexpect update on Yarn.lock during auto version upgrade #3856

• Validate POM Files before pushing to maven repo (nexus).
  Related Issue:
  Verify the artifacts in maven nexus releases repo. #2928

• Distribution build failure AUTOCUT issues do not provide detail information
  Related Issue: [Bug]: Distribution build failure GH issues do not provide detail information #4155

• Smaller Docker Images of OpenSearch and OpenSearch Dashboards.
  Related Issue: [META] Reduce docker image sizes for OpenSearch and OpenSearch Dashboards #3853

• RPM/DEB permissions update.
  Related Issue: Defaut config permission too relaxed in deb and rpm packages #3815

• Move the logic that is used for build which is now part of groovy libs to python. Move the validation groovy code to the validation python workflow.
  Related Issue: [Bug]: Unify code for all things that are used for a build #2609

• AUTOCUT issues does not honor windows failure [Bug]: Created AUTOCUT issues does not honor windows failure #4314

Documentation

• GitHub Wiki for OpenSearch Build repo.
  Related Issue: [Documentation] Have a better documentation and way to update and maintain it regularly #3936

• Release templates
  As a release manager every time one has to type and update the release issue, it would be easy if we have any templates (like RC is ready, Integ tests failing, starting the release process, pending version increment PR’s etc) to use them to update the release issue. An automation later that should auto tag the repo owners/maintainers.

[zelinh]

[Triage] We will be looking into this issue for enhancement.