Switch to AWS CI/CD #1356

andreyvelich · 2020-10-16T02:04:37Z

Related: #1332.
I will debug the infra in this PR.

I also made few changes to improve CI/CD quality.

/cc @gaocegege @johnugeorge
/cc @Jeffwan @PatrickXYS @jlewi @Bobgy

kubeflow-bot · 2020-10-16T02:04:45Z

This change is

PatrickXYS · 2020-10-16T02:41:13Z

test/scripts/v1beta1/create-cluster.sh

+    --node-type m5.xlarge \
+    --nodes 2 \
+    --nodes-min 1 \
+    --nodes-max 3


Can we use managed nodegroup?

Check here: https://github.com/kubeflow/testing/blob/master/images/aws-scripts/create-eks-cluster.sh#L36

I think I will use your script to create EKS cluster: https://github.com/kubeflow/testing/blob/master/images/aws-scripts/create-eks-cluster.sh. Since we clone kubeflow/testing repo to testing NFS, it should be possible.
What do you think @PatrickXYS ?

I think I will use your script to create EKS cluster

Yes we can, reuse common script makes a lot of sense to me.

PatrickXYS · 2020-10-16T16:43:49Z

/test ?

k8s-ci-robot · 2020-10-16T16:43:53Z

@PatrickXYS: The following commands are available to trigger jobs:

/test kubeflow-katib-presubmit

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

PatrickXYS · 2020-10-16T18:30:11Z

Pasted from #1357

The reason that plugin trigger can not trigger prow test:

{"author":"PatrickXYS","component":"hook","error":"error checking trust of PatrickXYS: error in IsCollaborator: return code not 2XX: 403 Forbidden","event-GUID":"16d24480-0fdd-11eb-911f-618d74ee83e5","event-type":"issue_comment","file":"prow/hook/events.go:425","func":"k8s.io/test-infra/prow/hook.(*Server).handleGenericComment.func1","level":"error","msg":"Error handling GenericCommentEvent.","org":"kubeflow","plugin":"trigger","pr":1357,"repo":"katib","severity":"error","time":"2020-10-16T18:26:29Z","url":"https://github.com/kubeflow/katib/pull/1357#issuecomment-710377544"}

It's caused by IsCollaborrator error on aws-kf-ci-bot,

the reason k8s-ci-bot works fine is k8s-ci-bot is admin of kubeflow org, so it can fetch org list to validate if member is Collaborator

I'm guessing only give write access is not enough, need to investigate

Pasted Prow Plugin trigger definition:

The trigger plugin starts tests in reaction to commands and pull request events. 

It is responsible for ensuring that test jobs are only run on trusted PRs. 

A PR is considered trusted if the author is a member of the 'trusted organization' for the repository or if such a member has left an '/ok-to-test' command on the PR.

Trigger starts jobs automatically when a new trusted PR is created or when an untrusted PR becomes trusted, but it can also be used to start jobs manually via the '/test' command.

The '/retest' command can be used to rerun jobs that have reported failure.

andreyvelich · 2020-10-16T18:31:51Z

@PatrickXYS Should we remove access k8s-ci-bot to Katib repo?
Or in that case, tide also doesn't work?

PatrickXYS · 2020-10-16T18:40:45Z

Removing kf-ci-bot from katib might be un-acceptable by google folks.

As we discussed in the proposal, Google Prow owns Tide functionality, and AWS Prow owns running tests and report PR status.

This should be one of the potential issue with mixed Prow, I'll keep you updated with the progress

/cc @andreyvelich

k8s-ci-robot · 2020-10-16T18:40:46Z

@PatrickXYS: GitHub didn't allow me to request PR reviews from the following users: andreyvelich.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Removing kf-ci-bot from katib might be un-acceptable by google folks.

As we discussed in the proposal, Google Prow owns Tide functionality, and AWS Prow owns running tests and report PR status.

This should be one of the potential issue with mixed Prow, I'll keep you updated with the progress

/cc @andreyvelich

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Jeffwan · 2020-10-16T20:01:13Z

I think the problem it won't be enable to fetch org member list to check collaborators.

Let's try to see if bot can get read access to entire org.

@PatrickXYS: GitHub didn't allow me to request PR reviews from the following users: andreyvelich.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

em.. This should be unexpected @andreyvelich is definitely a member. Anything wrong with the upstream prow because of new prow?

PatrickXYS · 2020-10-16T20:47:45Z

and authors cannot review their own PRs.

@Jeffwan This is expected, because @andreyvelich can't review his own PR.

aws-kf-ci-bot · 2020-10-16T21:33:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Jeffwan · 2020-10-16T21:35:18Z

@PatrickXYS seems bot does have access to the repo. However, I didn't see it can add label. approve label is added by k8s-ci-bit. Can you check backend logs?

Jeffwan · 2020-10-16T21:36:30Z

and authors cannot review their own PRs.

@Jeffwan This is expected, because @andreyvelich can't review his own PR.

I see. my careless. I thought it's failed to assign.

Jeffwan · 2020-10-16T21:36:43Z

/assign @andreyvelich

andreyvelich · 2020-10-23T14:47:16Z

/retest

PatrickXYS · 2020-10-23T16:58:08Z

/test ?

aws-kf-ci-bot · 2020-10-23T16:58:11Z

@PatrickXYS: The following commands are available to trigger jobs:

/test kubeflow-katib-presubmit-e2e

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2020-10-23T16:58:12Z

@PatrickXYS: The following commands are available to trigger jobs:

/test kubeflow-katib-presubmit

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

PatrickXYS · 2020-10-23T17:01:18Z

/test kubeflow-katib-presubmit

aws-kf-ci-bot · 2020-10-23T17:01:22Z

@PatrickXYS: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test kubeflow-katib-presubmit-e2e

Use /test all to run all jobs.

In response to this:

/test kubeflow-katib-presubmit

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

PatrickXYS · 2020-10-23T17:05:43Z

/lgtm

andreyvelich · 2020-10-24T03:19:25Z

@johnugeorge @gaocegege I was thinking about temporary solution for the registry.

Actually, it is easier to use just another workflow-v1beta1-release workflow during post-submit for us.
This workflow just pushes images to kubeflow-images-public and doesn't deploy any clusters.
In that case, we don't need to change website/manifests/katib repo's docs with temporary container registry.
After AWS release process implementation, we will directly switch registry to the new one.

What do you think?

k8s-ci-robot · 2020-10-24T06:13:10Z

@andreyvelich: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
kubeflow-katib-presubmit	`de3c82b`	link	`/test kubeflow-katib-presubmit`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

jlewi · 2020-10-26T13:32:59Z

Actually, it is easier to use just another workflow-v1beta1-release workflow during post-submit for us.
This workflow just pushes images to kubeflow-images-public

How will these jobs have access to kubeflow-images-public?

andreyvelich · 2020-10-26T14:14:35Z

@jlewi Using kubeflow-ci service account. This is just shor-term solution, until we implement new release process in AWS.
/cc @gaocegege @johnugeorge

johnugeorge · 2020-10-26T18:32:09Z

/lgtm

andreyvelich added 2 commits October 16, 2020 02:59

Add changes for AWS test infra

a1bbaa1

Remove comment from resume e2e

faa7f4c

k8s-ci-robot requested review from Bobgy, gaocegege, Jeffwan, jlewi, johnugeorge and PatrickXYS October 16, 2020 02:04

k8s-ci-robot added the do-not-merge/work-in-progress label Oct 16, 2020

k8s-ci-robot added the size/XL label Oct 16, 2020

kubeflow deleted a comment from kubeflow-bot Oct 16, 2020

Change worker image

1058b66

PatrickXYS reviewed Oct 16, 2020

View reviewed changes

Refactor e2e test script

0e38b84

k8s-ci-robot added size/XXL approved and removed size/XL labels Oct 16, 2020

k8s-ci-robot assigned johnugeorge Oct 23, 2020

k8s-ci-robot added the lgtm label Oct 23, 2020

k8s-ci-robot removed the lgtm label Oct 23, 2020

k8s-ci-robot assigned PatrickXYS Oct 23, 2020

k8s-ci-robot added lgtm and removed lgtm labels Oct 23, 2020

andreyvelich added the lgtm label Oct 23, 2020

andreyvelich force-pushed the aws-test-infra branch from ccb48bc to ce654af Compare October 23, 2020 18:52

k8s-ci-robot removed the lgtm label Oct 23, 2020

andreyvelich added the lgtm label Oct 23, 2020

Add release workflow

de3c82b

k8s-ci-robot removed the lgtm label Oct 24, 2020

andreyvelich mentioned this pull request Oct 26, 2020

Katib 0.10 v1beta1 release kubeflow/manifests#1593

Merged

1 task

k8s-ci-robot added the lgtm label Oct 26, 2020

k8s-ci-robot merged commit 64be948 into kubeflow:master Oct 26, 2020

andreyvelich mentioned this pull request Nov 3, 2020

Remove Kubeflow Katib from post-submit jobs kubernetes/test-infra#19818

Merged

andreyvelich mentioned this pull request Nov 13, 2020

Long term solution for AutoML CI/CD test infrastructure #1332

Closed

andreyvelich deleted the aws-test-infra branch October 6, 2021 00:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to AWS CI/CD #1356

Switch to AWS CI/CD #1356

andreyvelich commented Oct 16, 2020

kubeflow-bot commented Oct 16, 2020 •

edited by jlewi

Loading

PatrickXYS Oct 16, 2020

andreyvelich Oct 16, 2020 •

edited

Loading

PatrickXYS Oct 16, 2020 •

edited

Loading

PatrickXYS commented Oct 16, 2020

k8s-ci-robot commented Oct 16, 2020

PatrickXYS commented Oct 16, 2020 •

edited

Loading

andreyvelich commented Oct 16, 2020 •

edited

Loading

PatrickXYS commented Oct 16, 2020 •

edited

Loading

k8s-ci-robot commented Oct 16, 2020

Jeffwan commented Oct 16, 2020 •

edited

Loading

PatrickXYS commented Oct 16, 2020

aws-kf-ci-bot commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

andreyvelich commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

aws-kf-ci-bot commented Oct 23, 2020

k8s-ci-robot commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

aws-kf-ci-bot commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

andreyvelich commented Oct 24, 2020 •

edited

Loading

k8s-ci-robot commented Oct 24, 2020

jlewi commented Oct 26, 2020

andreyvelich commented Oct 26, 2020 •

edited

Loading

johnugeorge commented Oct 26, 2020

Switch to AWS CI/CD #1356

Switch to AWS CI/CD #1356

Conversation

andreyvelich commented Oct 16, 2020

kubeflow-bot commented Oct 16, 2020 • edited by jlewi Loading

PatrickXYS Oct 16, 2020

Choose a reason for hiding this comment

andreyvelich Oct 16, 2020 • edited Loading

Choose a reason for hiding this comment

PatrickXYS Oct 16, 2020 • edited Loading

Choose a reason for hiding this comment

PatrickXYS commented Oct 16, 2020

k8s-ci-robot commented Oct 16, 2020

PatrickXYS commented Oct 16, 2020 • edited Loading

andreyvelich commented Oct 16, 2020 • edited Loading

PatrickXYS commented Oct 16, 2020 • edited Loading

k8s-ci-robot commented Oct 16, 2020

Jeffwan commented Oct 16, 2020 • edited Loading

PatrickXYS commented Oct 16, 2020

aws-kf-ci-bot commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

Jeffwan commented Oct 16, 2020

andreyvelich commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

aws-kf-ci-bot commented Oct 23, 2020

k8s-ci-robot commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

aws-kf-ci-bot commented Oct 23, 2020

PatrickXYS commented Oct 23, 2020

andreyvelich commented Oct 24, 2020 • edited Loading

k8s-ci-robot commented Oct 24, 2020

jlewi commented Oct 26, 2020

andreyvelich commented Oct 26, 2020 • edited Loading

johnugeorge commented Oct 26, 2020

kubeflow-bot commented Oct 16, 2020 •

edited by jlewi

Loading

andreyvelich Oct 16, 2020 •

edited

Loading

PatrickXYS Oct 16, 2020 •

edited

Loading

PatrickXYS commented Oct 16, 2020 •

edited

Loading

andreyvelich commented Oct 16, 2020 •

edited

Loading

PatrickXYS commented Oct 16, 2020 •

edited

Loading

Jeffwan commented Oct 16, 2020 •

edited

Loading

andreyvelich commented Oct 24, 2020 •

edited

Loading

andreyvelich commented Oct 26, 2020 •

edited

Loading