[Cherry Pick] Enable canary report generation #634

jsitu777 · 2023-03-24T22:59:10Z

-enable canary report generation after each canary run.

…s/kubeflow-manifests into release-v1.6.1-aws-b1.0.1

git add tests/canary/canary.buildspec.yaml git push origin enable-xml-report q

… cloudwatch metrics

…f docker cp fails

surajkota

approving with comments, please address and merge

surajkota · 2023-03-30T19:48:22Z

tests/canary/scripts/push_stats_to_cloudwatch.py

+    cw_client = boto3.client("cloudwatch")
+    project_name = "CodeBuild-Run-All-Tests"
+
+    cluster_name = os.getenv("CLUSTER_NAME")


nit: remove

surajkota · 2023-03-30T19:49:57Z

tests/canary/scripts/run_test.sh

+  echo "Pushing Codebuild stats to Cloudwatch."
+  python ../canary/scripts/push_stats_to_cloudwatch.py
+}
+trap push_to_cloudwatch EXIT


move this code above line 13 since failure can happen in any command

surajkota · 2023-03-30T19:50:16Z

tests/e2e/fixtures/installation.py

@@ -63,7 +63,6 @@ def on_create():
        install_kubeflow(installation_option, deployment_option, cluster)

    def on_delete():
-


nit: revert? or coming from black

-enable canary report generation after each canary run.

This reverts commit 632ec70.

-enable canary report generation after each canary run.

-Merge release branch PR to main #634

-enable canary report generation after each canary run.

This reverts commit 632ec70.

-enable canary report generation after each canary run.

This reverts commit 632ec70.

-enable canary report generation after each canary run.

This reverts commit 632ec70.

-enable canary report generation after each canary run.

This reverts commit 632ec70.

-Merge release branch PR to main awslabs#634

* Update website links to v1.6.1 (awslabs#487) Update website links to v1.6.1 * Update git clone command (awslabs#488) Update git clone command * Katib S3-only Helm Path cherrypick (awslabs#507) Cherry pick the helm path installation config fix for s3-only deployment. Co-authored-by: Pei Ran Li <prli@users.noreply.github.com> * Cherry-pick Kserve with IRSA and Notebook culling Doc (awslabs#512) * Cherry-pick Kserve with IRSA and Notebook culling Doc into v1.6.1 release * Cherrypick: Add permission configuration steps to SageMaker KFP docs (awslabs#506) (awslabs#528) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** - Adds required permission steps for using Sagemaker v1/v2 integration with KFP **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * New blog/workshop update - 1.6 (awslabs#533) By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * docsearch backport 1.6 (awslabs#534) docsearch first integration By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * 1.6 * helm provider * blueprints * fix oidc * Cherry-Pick terraform bug fixes (awslabs#590) **Which issue is resolved by this Pull Request:** **Description of your changes:** Update RDS version awslabs#584 Update AWS blueprints - gpu bug awslabs#516 **Testing:** Cognito-rds-s3 passed `6 passed, 13 warnings in 4594.63s (1:16:34)` rds-s3 passed `7 passed, 49 warnings in 4490.15s (1:14:50)` - [ ] Unit tests pass - [x] e2e tests pass By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Shadab Hussain <shadab.entrepreneur@outlook.com> Co-authored-by: Gerhard Häring <gh@ghaering.de> * Updating website to reflect latest version (awslabs#596) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Ubuntu <ubuntu@ip-172-31-58-39.us-west-2.compute.internal> * Cherry pick sagemaker fix and move of kfp SM test (awslabs#619) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** Cherry pick sagemaker fix and move of kfp SM test **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * fix secrets * [Cherry pick] Change Dex Service from Nodeport to ClusterIP (awslabs#630) **Description of your changes:** There are occasional times when installing dex we see the following error message: ``` The Service "dex" is invalid: spec.ports[0].nodePort: Invalid value: 32000: provided port is already allocated ``` It indicates that there is already a service using port 32000 on the same node trying to create the "dex" service. Dex service was modified to be exposed as ClusterIP instead of NodePort in the new release KF v1.7.0 along with Istio for security upgrade (kubeflow/manifests#2357) This is a mirror patch from the above PR to change Dex service from NodePort to ClusterIP to solve the error message seen. **Testing:** - [ ] Unit tests pass - [x] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * [Cherry-Pick] increase cert-manager wait time for kubeflow-issuer to be install (awslabs#632) Cherry pick this commit from main to temporarily solve: ``` : Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "[https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s](https://cert-manager-webhook.cert-manager.svc/mutate?timeout=10s)": context deadline exceeded ``` * variables * [Cherry Pick] Enable canary report generation (awslabs#634) -enable canary report generation after each canary run. * Update blog content (awslabs#638) **Description of your changes:** - Remove Trainium (pending future release) - Update link to AWS Docs (was localhost) - Update ordered list numbering **Testing:** - Tested local Hugo build By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Suraj Kota <surakota@amazon.com> * add pipeline pd * [Cherry-pick] Push Canary Success Rate to Cloudwatch Metric (awslabs#662) - cherry-pick pushing success_rate into cloudwatch metrics * Release v1.6.1 aws b1.0.2 (awslabs#666) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** Note: As a result of the depreciation of k8s.gcr.io, images have moved to registry.k8s.io. This only effects csi-secrets-driver for those who have installed either RDS-S3/Cognito-RDS-S3. To update your pulled image source run the following kubectl commands ``` kubectl set image daemonset/csi-secrets-store node-driver-registrar=registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0 -n kube-system kubectl set image daemonset/csi-secrets-store secrets-store=registry.k8s.io/csi-secrets-store/driver:v1.0.0-rc.1 -n kube-system kubectl set image daemonset/csi-secrets-store liveness-probe=registry.k8s.io/sig-storage/livenessprobe:v2.4.0 -n kube-system ``` **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * update website to release-v1.6.1-aws-b1.0.2 (awslabs#667) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** Points toml and documentation to latest release version **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * Update config.toml for v1.6.1 (awslabs#705) - change version from `latest` to `1.6` * [Cherry pick] Kserve Indentation doc fix v1.6.1 (awslabs#721) - cherry-pick for this PR awslabs#719 - Keep empty AWS credential strings * fix evictions * value needed for defaults * Website changes for release branch (awslabs#758) **Which issue is resolved by this Pull Request:** Resolves # **Description of your changes:** - Same as title By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. * refactor --------- Co-authored-by: Kartik Kalamadi <kalamadi@amazon.com> Co-authored-by: jsitu777 <59303945+jsitu777@users.noreply.github.com> Co-authored-by: Pei Ran Li <prli@users.noreply.github.com> Co-authored-by: ryansteakley <37981995+ryansteakley@users.noreply.github.com> Co-authored-by: Nadege PEPIN <5490706+npepin-hub@users.noreply.github.com> Co-authored-by: ananth102 <abashyam@amazon.com> Co-authored-by: Shadab Hussain <shadab.entrepreneur@outlook.com> Co-authored-by: Gerhard Häring <gh@ghaering.de> Co-authored-by: Ubuntu <ubuntu@ip-172-31-58-39.us-west-2.compute.internal> Co-authored-by: Kevin Hoyt <parkerkrhoyt@gmail.com> Co-authored-by: Suraj Kota <surakota@amazon.com>

jsitu777 added 17 commits March 21, 2023 15:58

increase cert-manager wait time for kubeflow-issuer to be install

8a2eefe

Merge branch 'release-v1.6.1-aws-b1.0.1' of https://github.com/awslab…

fbf91ac

…s/kubeflow-manifests into release-v1.6.1-aws-b1.0.1

enable canary report

0e973cf

modify container name

76f1bb6

change docker command

8c3662d

change docker command

3d97489

print out container file structure to debug

5e657e3

print out container file structure to debug

ae0609e

debug container

ebd48b4

fix canary.buildspec.yaml

8c6efd9

fix canary.buildspec.yaml

045b498

fix canary.buildspec.yaml

4ca26b4

push to ECR after cp

ff4c386

push to ECR after cp

b5f838d

run container in detached mode so it's still running after build

bc81c59

git add tests/canary/canary.buildspec.yaml git push origin enable-xml-report q

sleep 30s to ensure container still running

e4bd53b

remove sleep

77bd0a7

jsitu777 changed the title ~~Enable canary report generation~~ [Cherry-pick] Enable canary report generation Mar 25, 2023

jsitu777 added 9 commits March 28, 2023 17:07

push to cloudwatch

3be38a8

minor fix

147debd

bug fixes

02db0e1

rename test log to xml

fadb8e7

add dimensions to cloudwatch metrics

10350bf

use trap to make sure it python executes if pytest fails

76a15e4

use list_project for codebuild

a8891c1

fix datetime module error

09104dc

remove region as cloudwatch metric dimension

fc4cd2d

jsitu777 changed the title ~~[Cherry-pick] Enable canary report generation~~ Enable canary report generation Mar 29, 2023

jsitu777 changed the base branch from release-v1.6.1-aws-b1.0.1 to main March 29, 2023 23:16

jsitu777 changed the base branch from main to release-v1.6.1-aws-b1.0.0 March 29, 2023 23:16

jsitu777 added 8 commits March 29, 2023 19:16

add set -e in buildspec

bebf9b0

to ensure docker always exit with 0

350b322

bug fix

92dc0b2

remove || true after docker run, hard code codebuild project name for…

b81c41c

… cloudwatch metrics

restore kf installation and uninstallation in fixture

95ab554

test if cloudwatch can push metrics if test fails

3a61765

teest if can get env variable directly

25aa392

restore kf installation and notebook test

5e930d3

jsitu777 requested a review from surajkota March 30, 2023 16:45

jsitu777 added 2 commits March 30, 2023 09:57

add || true to docker cp command so next command will continue even i…

2e395b4

…f docker cp fails

rename cloudwatch metric namespace

fe09879

surajkota approved these changes Mar 30, 2023

View reviewed changes

move trap code at the beginning; remove getting cluster_name env

f703021

jsitu777 changed the title ~~Enable canary report generation~~ [Cherry Pick] Enable canary report generation Mar 30, 2023

jsitu777 merged commit 22ca52c into awslabs:release-v1.6.1-aws-b1.0.1 Mar 30, 2023

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Mar 30, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

632ec70

-enable canary report generation after each canary run.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Mar 30, 2023

Revert "[Cherry Pick] Enable canary report generation (awslabs#634)"

1a34003

This reverts commit 632ec70.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Mar 30, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

4208e96

-enable canary report generation after each canary run.

jsitu777 mentioned this pull request Mar 30, 2023

Enable canary report generation #643

Merged

jsitu777 added a commit that referenced this pull request Mar 30, 2023

Enable canary report generation (#643)

58b3041

-Merge release branch PR to main #634

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

70dab26

-enable canary report generation after each canary run.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

Revert "[Cherry Pick] Enable canary report generation (awslabs#634)"

f03d691

This reverts commit 632ec70.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

e14a654

-enable canary report generation after each canary run.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

Revert "[Cherry Pick] Enable canary report generation (awslabs#634)"

eb65731

This reverts commit 632ec70.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

43c4ef4

-enable canary report generation after each canary run.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

Revert "[Cherry Pick] Enable canary report generation (awslabs#634)"

02c2de6

This reverts commit 632ec70.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

[Cherry Pick] Enable canary report generation (awslabs#634)

deee51d

-enable canary report generation after each canary run.

jsitu777 added a commit to jsitu777/kubeflow-manifests that referenced this pull request Apr 12, 2023

Revert "[Cherry Pick] Enable canary report generation (awslabs#634)"

5d3b430

This reverts commit 632ec70.

ryansteakley pushed a commit to ryansteakley/kubeflow-manifests that referenced this pull request Apr 14, 2023

Enable canary report generation (awslabs#643)

a4c5c34

-Merge release branch PR to main awslabs#634

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cherry Pick] Enable canary report generation #634

[Cherry Pick] Enable canary report generation #634

jsitu777 commented Mar 24, 2023

surajkota left a comment

surajkota Mar 30, 2023

surajkota Mar 30, 2023

surajkota Mar 30, 2023

		@@ -63,7 +63,6 @@ def on_create():
		install_kubeflow(installation_option, deployment_option, cluster)

		def on_delete():

[Cherry Pick] Enable canary report generation #634

[Cherry Pick] Enable canary report generation #634

Conversation

jsitu777 commented Mar 24, 2023

surajkota left a comment

Choose a reason for hiding this comment

surajkota Mar 30, 2023

Choose a reason for hiding this comment

surajkota Mar 30, 2023

Choose a reason for hiding this comment

surajkota Mar 30, 2023

Choose a reason for hiding this comment