Add 1.1.1 issue template and input manifests (#1275)

* Add 1.1.1 related templates and manifests Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * More changes to template and bundle code to make build 1.1.1 opensearch runs Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add more changes Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add arm64 and docker build code Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore Jenkinsfile Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore Jenkinsfile Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore build from source python code Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add 1.1.1 related templates and manifests Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * More changes to template and bundle code to make build 1.1.1 opensearch runs Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add more changes Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add arm64 and docker build code Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore Jenkinsfile Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore Jenkinsfile Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Restore build from source python code Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Readd functional tests Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Switch checks of build from source to gradle related checks Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Remove checks for dashboards as it is not there yet Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Fix ci check link is not matching the new setup in s3 Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add more changes to resolve issues with templates and tests Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * More updates to the templates sent to owners Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>
opensearch-project · Dec 9, 2021 · 760b5f2 · 760b5f2
1 parent c3d9a85
commit 760b5f2
Show file tree

Hide file tree

Showing 15 changed files with 210 additions and 50 deletions.
diff --git a/.github/meta_issue_template/release-1.1.1-component-issues.md b/.github/meta_issue_template/release-1.1.1-component-issues.md
@@ -0,0 +1,56 @@
+This is a component issue for release 1.1.1.
+Coming from [release issue 1.1.1](https://github.com/opensearch-project/opensearch-build/issues/870), release version 1.1.1. Please follow the following checklist.
+
+<details><summary>How to use this component issue</summary>
+<p>
+
+## This Component Issue
+This component issue captures the state of the OpenSearch release, on component/plugin level, its assignee is responsible for driving the release of the component.  Please contact them or @mention them on this issue for help. 
+
+## Release Steps
+There are several steps to the release process, components that are behind present risk to the release.  Component owners resolve tasks on this ticket to communicate with the overall release owner.
+
+Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.
+
+You can find all the corresponding dates of each step in the release issue above.
+
+### What should I do if my plugin isn't making any changes?
+If including changes in this release, increment the version on 1.1 branch to `1.1.1` for Min/Core, and `1.1.1.0` for components. Otherwise, keep the version number unchanged for both.
+
+</p>
+</details>
+
+
+### You can find all the date in above issue 
+
+### Preparation
+
+- [ ] Assign this issue to a release owner.
+- [ ] All the tasks in this issue have been reviewed by the release owner.
+- [ ] Create, update, triage and label all features and issues targeted for this release with v1.1.1.
+
+
+### CI/CD
+
+- [ ] All code changes for 1.1.1 are complete.
+- [ ] Ensure working and passing CI.
+
+### Pre-Release
+
+- [ ] Confirm that all changes for 1.1.1 have been merged.
+- [ ] Complete integration and sanity tests, and update results in the comment, [example](https://github.com/opensearch-project/opensearch-build/issues/1118).
+- [ ] Find/fix bugs using latest tarball and docker image provided in meta issue.
+- [ ] Completed release candidate testing build #TBD.
+- [ ] All intermittent test failures have issues filed.
+
+### Release
+
+- [ ] Complete [documentation](https://github.com/opensearch-project/documentation-website).
+- [ ] Gather, review and publish release notes.
+- [ ] Verify all issued labeled for this release are closed or labelled for the next release.
+
+### Post Release
+
+- [ ] Create [a release tag](https://github.com/opensearch-project/.github/blob/main/RELEASING.md#tagging).
+- [ ] Suggest improvements to [this template](https://github.com/opensearch-project/opensearch-build/blob/main/.github/ISSUE_TEMPLATE/release_template.md).
+- [ ] Conduct a retrospect, and publish its results.
diff --git a/.github/meta_issue_template/release-1.3.0-component-issues.md b/.github/meta_issue_template/release-1.3.0-component-issues.md
@@ -0,0 +1,59 @@
+This is a component issue for release 1.3.0.
+Coming from [release issue 1.3.0](https://github.com/opensearch-project/opensearch-build/issues/870), release version 1.3.0. Please follow the following checklist.
+
+<details><summary>How to use this component issue</summary>
+<p>
+
+## This Component Issue
+This component issue captures the state of the OpenSearch release, on component/plugin level, its assignee is responsible for driving the release of the component.  Please contact them or @mention them on this issue for help. 
+
+## Release Steps
+There are several steps to the release process, components that are behind present risk to the release.  Component owners resolve tasks on this ticket to communicate with the overall release owner.
+
+Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.
+
+You can find all the corresponding dates of each step in the release issue above.
+
+### What should I do if my plugin isn't making any changes?
+If including changes in this release, increment the version on 1.1 branch to `1.1.1` for Min/Core, and `1.1.1.0` for components. Otherwise, keep the version number unchanged for both.
+
+</p>
+</details>
+
+
+### You can find all the date in above issue 
+
+### Preparation
+
+- [ ] Assign this issue to a release owner.
+- [ ] Finalize scope and feature set and update [the Public Roadmap](https://github.com/orgs/opensearch-project/projects/1).
+- [ ] All the tasks in this issue have been reviewed by the release owner.
+- [ ] Create, update, triage and label all features and issues targeted for this release with v1.3.0.
+
+
+### CI/CD
+
+- [ ] All code changes for 1.3.0 are complete.
+- [ ] Ensure working and passing CI.
+- [ ] Re(add) this repo to the (if not exist) [opensearch-dashboards input manifest](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/1.3.0/opensearch-dashboards-1.3.0.yml).
+
+### Pre-Release
+
+- [ ] Update your branch in the [opensearch-dashboards input manifest](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/1.3.0/opensearch-dashboards-1.3.0.yml).
+- [ ] Confirm that all changes for 1.1.1 have been merged.
+- [ ] Complete integration tests, and update results in the comment, [example](https://github.com/opensearch-project/opensearch-build/issues/1118).
+- [ ] Find/fix bugs using latest tarball and docker image provided in meta issue.
+- [ ] Completed release candidate testing build #TBD.
+- [ ] All intermittent test failures have issues filed.
+
+### Release
+
+- [ ] Complete [documentation](https://github.com/opensearch-project/documentation-website).
+- [ ] Gather, review and publish release notes.
+- [ ] Verify all issued labeled for this release are closed or labelled for the next release.
+
+### Post Release
+
+- [ ] Create [a release tag](https://github.com/opensearch-project/.github/blob/main/RELEASING.md#tagging).
+- [ ] Suggest improvements to [this template](https://github.com/opensearch-project/opensearch-build/blob/main/.github/ISSUE_TEMPLATE/release_template.md).
+- [ ] Conduct a retrospect, and publish its results.
diff --git a/manifests/1.1.1/opensearch-1.1.1.yml b/manifests/1.1.1/opensearch-1.1.1.yml
@@ -1,73 +1,74 @@
 ---
 schema-version: "1.0"
+ci:
+  image:
+     name: "opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211028"
 build:
   name: OpenSearch
   version: 1.1.1
   patches: 
     - 1.1.0
 components:
   - name: OpenSearch
-    repository: https://github.com/opensearch-project/OpenSearch.git
-    ref: "1.1"
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
-      - gradle:publish
-      - gradle:properties:version
+      - manifest:component
   - name: common-utils
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: job-scheduler
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
+  - name: dashboards-reports
+    repository: https://github.com/opensearch-project/dashboards-reports.git
+    working_directory: reports-scheduler
+    ref: "1.1"
+    platforms:
+      - linux
+    checks:
+      - gradle:publish
+      - gradle:properties:version
   - name: sql
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: alerting
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: security
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: cross-cluster-replication
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
-    checks:
-      - manifest:component
-  - name: performance-analyzer-rca
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: performance-analyzer
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: index-management
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: k-NN
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
     platforms:
-      - darwin
       - linux
   - name: anomaly-detection
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: asynchronous-search
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
-    checks:
-      - manifest:component
-  - name: dashboards-reports
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
   - name: dashboards-notebooks
-    dist: https://ci.opensearch.org/ci/dbc/builds/1.1.0/405
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930
     checks:
       - manifest:component
diff --git a/manifests/1.1.1/opensearch-dashboards-1.1.1.yml b/manifests/1.1.1/opensearch-dashboards-1.1.1.yml
@@ -0,0 +1,38 @@
+---
+schema-version: "1.0"
+ci:
+  image:
+     name: "opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211028"
+build:
+  name: OpenSearch Dashboards
+  version: 1.1.1
+  patches:
+    - 1.1.0
+components:
+  - name: reportsDashboards
+    repository: https://github.com/opensearch-project/dashboards-reports
+    ref: "1.1"
+    working_directory: "dashboards-reports"
+    platforms:
+      - linux
+  - name: OpenSearch-Dashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: alertingDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: securityDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: indexManagementDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: queryWorkbenchDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: notebooksDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: ganttChartDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: traceAnalyticsDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: anomalyDetectionDashboards
+    dist: https://ci.opensearch.org/ci/dbc/bundle-build-dashboards/1.1.0/20210930
+  - name: functionalTestDashboards
+    repository: https://github.com/opensearch-project/opensearch-dashboards-functional-test.git
+    ref: "main"
diff --git a/src/ci_workflow/ci_check_manifest_component.py b/src/ci_workflow/ci_check_manifest_component.py
@@ -18,7 +18,9 @@ def __init__(self, component, url):
 
     def check(self):
         for architecture in BuildArgs.SUPPORTED_ARCHITECTURES:
-            url = "/".join([self.component.dist, architecture, "manifest.yml"])
+            # Since we only have 'linux' builds now we hard code it to 'linux'
+            # Once we have all platform builds on S3 we can then add a second loop for 'BuildArgs.SUPPORTED_PLATFORMS'
+            url = "/".join([self.component.dist, "linux", architecture, "builds", self.target.name, "manifest.yml"])
             self.build_manifest = BuildManifest.from_url(url)
             if self.component.name in self.build_manifest.components:
                 logging.info(f"Found {self.component.name} in {url}.")

diff --git a/src/ci_workflow/ci_input_manifest.py b/src/ci_workflow/ci_input_manifest.py
@@ -19,7 +19,7 @@ def __init__(self, file, args):
 
     def __check__(self):
 
-        target = CiTarget(version=self.manifest.build.version, snapshot=self.args.snapshot)
+        target = CiTarget(version=self.manifest.build.version, name=self.manifest.build.filename, snapshot=self.args.snapshot)
 
         with TemporaryDirectory(keep=self.args.keep, chdir=True) as work_dir:
             logging.info(f"Sanity-testing in {work_dir.name}")

diff --git a/src/ci_workflow/ci_target.py b/src/ci_workflow/ci_target.py
@@ -7,10 +7,12 @@
 
 class CiTarget:
     version: str
+    name: str
     snapshot: bool
 
-    def __init__(self, version, snapshot=True):
+    def __init__(self, version, name, snapshot=True):
         self.version = version
+        self.name = name
         self.snapshot = snapshot
 
     @property

diff --git a/tests/tests_ci_workflow/test_ci_check_gradle_dependencies.py b/tests/tests_ci_workflow/test_ci_check_gradle_dependencies.py
@@ -24,7 +24,7 @@ def __mock_dependencies(self, props="", snapshot=False, gradle_project=None):
         return TestCiCheckGradleDependencies.DummyDependencies(
             component=MagicMock(),
             git_repo=git_repo,
-            target=CiTarget(version="1.1.0", snapshot=snapshot),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=snapshot),
             args=gradle_project,
         )
 

diff --git a/tests/tests_ci_workflow/test_ci_check_gradle_dependencies_opensearch.py b/tests/tests_ci_workflow/test_ci_check_gradle_dependencies_opensearch.py
@@ -22,7 +22,7 @@ def __mock_check(self, props=None):
             return CiCheckGradleDependenciesOpenSearchVersion(
                 component=MagicMock(),
                 git_repo=MagicMock(),
-                target=CiTarget(version="1.1.0", snapshot=True),
+                target=CiTarget(version="1.1.0", name="opensearch", snapshot=True),
                 args=None,
             )
 
@@ -52,7 +52,7 @@ def test_executes_gradle_command(self):
         check = CiCheckGradleDependenciesOpenSearchVersion(
             component=MagicMock(),
             git_repo=MagicMock(),
-            target=CiTarget(version="1.1.0", snapshot=True),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=True),
             args=None,
         )
         check.git_repo.output.assert_called_once_with(
@@ -63,7 +63,7 @@ def test_executes_gradle_command_with_arg(self):
         check = CiCheckGradleDependenciesOpenSearchVersion(
             component=MagicMock(),
             git_repo=MagicMock(),
-            target=CiTarget(version="1.1.0", snapshot=True),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=True),
             args="plugin",
         )
         check.git_repo.output.assert_called_once_with(

diff --git a/tests/tests_ci_workflow/test_ci_check_gradle_properties.py b/tests/tests_ci_workflow/test_ci_check_gradle_properties.py
@@ -23,7 +23,7 @@ def test_executes_gradle_properties(self):
         TestCiCheckGradleProperties.DummyProperties(
             component=MagicMock(),
             git_repo=git_repo,
-            target=CiTarget(version="1.1.0", snapshot=False),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=False),
         )
 
         git_repo.output.assert_called_once_with("./gradlew properties -Dopensearch.version=1.1.0 -Dbuild.snapshot=false")
@@ -35,7 +35,7 @@ def test_executes_gradle_properties_snapshot(self):
         TestCiCheckGradleProperties.DummyProperties(
             component=MagicMock(),
             git_repo=git_repo,
-            target=CiTarget(version="1.1.0", snapshot=True),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=True),
         )
 
         git_repo.output.assert_called_once_with("./gradlew properties -Dopensearch.version=1.1.0-SNAPSHOT -Dbuild.snapshot=true")
diff --git a/tests/tests_ci_workflow/test_ci_check_gradle_properties_version.py b/tests/tests_ci_workflow/test_ci_check_gradle_properties_version.py
@@ -20,7 +20,7 @@ def __mock_check(self, props=None, component=None, snapshot=True):
             return CiCheckGradlePropertiesVersion(
                 component=component or MagicMock(),
                 git_repo=MagicMock(),
-                target=CiTarget(version="1.1.0", snapshot=snapshot),
+                target=CiTarget(version="1.1.0", name="opensearch", snapshot=snapshot),
             )
 
     def test_has_version(self):

diff --git a/tests/tests_ci_workflow/test_ci_check_gradle_publish_to_maven_local.py b/tests/tests_ci_workflow/test_ci_check_gradle_publish_to_maven_local.py
@@ -16,7 +16,7 @@ def test_executes_gradle_command(self):
         check = CiCheckGradlePublishToMavenLocal(
             component=MagicMock(),
             git_repo=MagicMock(),
-            target=CiTarget(version="1.1.0", snapshot=False),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=False),
         )
         check.check()
         check.git_repo.execute.assert_called_once_with("./gradlew publishToMavenLocal -Dopensearch.version=1.1.0 -Dbuild.snapshot=false")
@@ -25,7 +25,7 @@ def test_executes_gradle_command_snapshot(self):
         check = CiCheckGradlePublishToMavenLocal(
             component=MagicMock(),
             git_repo=MagicMock(),
-            target=CiTarget(version="1.1.0", snapshot=True),
+            target=CiTarget(version="1.1.0", name="opensearch", snapshot=True),
         )
         check.check()
         check.git_repo.execute.assert_called_once_with("./gradlew publishToMavenLocal -Dopensearch.version=1.1.0-SNAPSHOT -Dbuild.snapshot=true")