[FEATURE] OpenSearch Project Central Release Dashboard

[prudhvigodithi]

Feature request Details

Use the Metrics Dashboard to track the OpenSearch release progress.

Once all the release details collected with a proper data model these release details can be visualized in the OpenSearch Dashboards which can be utilized by the release manager to oversee the release and serve as a reference point for the audience participating in the release.

What solution would you like?

While we have this Dashboard that can be used during a release that would help the release manager, the following improvements can be added to make the dashboard more informative and can be used to drive the release.

During the release

• The release owner for a specific component.
• The release component GitHub issues with no user assigned.
• The triggered RC build details.
• RC build details.
• Integration test results and links to download the logs.
• CVE report for the generated RC.
• Filter the RC build integration test results per component and architecture.
• The AUTOCUT issues per repo.
• Release label issues open and closed.
• Release label pulls open and merged.
• Components with no release notes.
• Components with no release branch.
• Release branch code coverage scores.
• Pending version increment repos for current, past and upcoming release, should be able to filter by release.
• Based on the triggered RC build, automation to add a comment in GitHub similar to [RELEASE] Release version 2.15.0 opensearch-build#4681 (comment).
• Based on the triggered RC integration test build, automation to add a comment in GitHub similar to [RELEASE] Release version 2.15.0 opensearch-build#4681 (comment).
• Issues with the "needs-doc" label and if there is a corresponding open documentation issue(true/false).
• Metrics to track the integration test failures at the tests level.
• Ability to filter by component, component_repo and component_repo_url. For OpenSearch Dashboard components this is important as component name and repo are different (example the component is ganttChartDashboards but the repo is dashboards-visualizations) and for OpenSearch-Dashboards core the tests are executed as multiple ci groups. So having filter by component, component_repo and component_repo_url should get the right set of documents.

Post Release

• Mechanism to auto update the ReleaseInputs, the release state.

Release Retro Meetings

The following data can be utilized in retrospective meetings to review the data points between the RC builds and to develop an action plan for the upcoming releases.

• Number of RC’s created for a release: Count metric.
• Trend graph of number of RC's created per release.
• Trend graph of number of plugin commitID changes per RC for a release.
• Add a label retro with release version where dashboard shows these issue links and one can discuss these issues in retro meeting.
• RC build number/link for both OS and OSD.
• Data table for the commit ID used for a given RC build.
• Show code quality and coverage for RC commit in data table for each component part of the release.
• Show integration test results for each RC, failed and passed components and details (in data table), filter per status failed or passed and per architecture.
• Have a mechanism to compare 2 RC’s.
  • Compare the number of commits between RC’s.
  • Plugin names that had a commit change.
  • Compare the integration test results between RC’s.
  • Compare the number of lines added per commit between RC’s.
  • Compare the code quality between 2 RC commits.
• Have all the above metrics filter per RC/RC’s and release version
• Compare Performance test results between RC’s.
• CVE’s between RC’s.
• Documentation PR’s/Changes between RC’s.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[prudhvigodithi]

[Triage]
Adding @dblock @getsaurabh02 to please take a look and provide some inputs.
Thank you

[peterzhuamazon]

Adding a few more to the list:

• Linking release owner internal alias, public slack username, github id all to one place so it is easier to find SME.
• Matching new feature with the corresponding doc issue and PRs, instead of asking tech writer to manually find which feature is missing docs.
• Test-reruns and how to link them to existing RCs.
• All changes get committed between RCs.
• Potential / important issues and campaigns that would block the release.
• Retro Items linking from GitHub Projects to Release Dashboards.

More to come.

Thanks.

[peterzhuamazon]

Update:

It is easier to use the report.yml provided during each integTest runs for indexing data for metrics:
opensearch-project/alerting#1598

Therefore I will go ahead and add a few extra information in the report workflow, so @prudhvigodithi can use them directly instead of taking from Jenkins runs:

1. architecture
2. distribution
3. is_rc
4. rc_number
5. buildID
6. platform
7. more......

Thanks.

---

PRs:

• Add 1.1 version of test report yml with bundle and rc information opensearch-build#4887
• Add 1.1 test-report.yml support to Jenkins lib opensearch-build-libraries#470
• Update integTest with 1.1 test-report.yml libs 6.8.3 version opensearch-build#4956
• Remove old args check on publishIntegTestResults opensearch-build-libraries#489
• Update integTest with lib version 6.8.4 opensearch-build#4961

[peterzhuamazon]

The code is not fully fixed:
https://github.com/opensearch-project/opensearch-build-libraries/blob/main/vars/publishIntegTestResults.groovy#L25-L33

[peterzhuamazon]

New functions required:

• Test stdout/stderr
• Component repo link
• [FEATURE] Metrics to track the integration test failures at the tests level  #68

[prudhvigodithi]

New functions required:

• Test stdout/stderr
• Component repo link

Thanks @peterzhuamazon, the Component repo name should be good and we can generate the repo URL, example https://github.com/opensearch-project/{repo_name}.

[peterzhuamazon]

New functions required:

• Test stdout/stderr
• Component repo link

Thanks @peterzhuamazon, the Component repo name should be good and we can generate the repo URL, example https://github.com/opensearch-project/{repo_name}.

PR:

• Add test_stdout and test_stderr path/url for test-report.yml opensearch-build#5001
• Add component repository key value pair to report yml opensearch-build#5027

[prudhvigodithi]

Added 2 new visualizations for release metrics

• Compares the Commits between the RC’s.
• Identifies the repos with changes between the RC’s.
• We should be able to filter per repo or do a comparison between one to many RC’s.