AWS Init Support for Kuberflow/Testing

[PatrickXYS]

Which issue is resolved by this Pull Request:
Resolves #748

Description of your changes:

1. Modify run_workflows.sh and run_e2e_workflow.py to make them AWS compatible.
2. Add Dockerfile.py3.aws for AWS ECR Image.
3. Add images/aws-scripts folder to place AWS related scripts.
4. Add cloudprovider/aws folder to place AWS related python files.
   Since we added import boto3 in util.py, GCP also need to install boto3 in the Dockerfile. Thus, modifies Dockerfile and Dockerfile.py3

[kubeflow-bot]

This change is 

[PatrickXYS]

/cc @Jeffwan @jlewi @rmgogogo

[andreyvelich]

@PatrickXYS Thanks a lot for doing this!
I left few comments

[PatrickXYS]

Per presubmit job logs:

...
File "/src/kubeflow/testing/py/kubeflow/testing/util.py", line 27, in <module> import boto3 ImportError: No module named boto3
...

To make kubeflow-testing-presubmit succeed, we need GCP folks to first send PR to build new container image with boto3 installed.

This is TBD for now. Will pick this up once we all agree to move forward and merge this PR.

[jlewi]

To make kubeflow-testing-presubmit succeed, we need GCP folks to first send PR to build new container image with boto3 installed.

The folks listed in: https://github.com/kubeflow/internal-acls/blob/master/google_groups/ci-team.members.txt

Should be able to help with that

[jlewi]

Although

File "/src/kubeflow/testing/py/kubeflow/testing/util.py", line 27, in <module> import boto3 ImportError: No module named boto3
...

Can we avoid this by organizing the code so that we don't have to pull in platform/cloud specific libraries when not running on that platform?

Could we create different subpackages for different clouds that can be selectively imported?

e.g. 

py/kubeflow/testing/aws

This will also make code review easier since we can have separate OWNERs files

[PatrickXYS]

Per the discussion in Kubeflow Test-infra on AWS, if Google will host Prow layer, and AWS host CI layer, there's no need to create and upload started.json + finished.json to S3.

If so, the step which differs from GCP CI-infra, is:

1. Configure AWS credential and load AWS CI cluster's kubeconfig
2. Ksonnet generate and apply Argo Workflow to AWS CI cluster.

Besides that, we need to create secret in GCP Prow cluster to store AWS credentials.

I misunderstood at the beginning.

This is summary of the proposal:

From AWS side, we have:

1. Bot account -> be granted with permissions to some repos.
2. Bot Webhook -> set up in some repo’s setting and be able to receive any github event
3. Prow cluster -> everything without Tide installed.
4. CI cluster -> Installed with Argo

In any of PR, specific tests are running by AWS bot, and Tide is delegated by GCP bot.

[PatrickXYS]

@Jeffwan @jlewi @andreyvelich

I cleaned up the code and nothing blocks the PR for now. This will be shipped as part of POC work.

Can you take another look?

[andreyvelich]

@PatrickXYS Since we create another util for AWS, my suggestion is: left only functions that we are using.

So you can delete everything except these functions: upload_to_s3, aws_configure_credential, load_kube_config, upload_file_to_s3, run, to_s3_uri, split_s3_uri.
Later we can add more features, if it is needed.

What do you think?

[jlewi]

On my end
/lgtm

@Bobgy Thoughts?

[Jeffwan]

/lgtm

[andreyvelich]

Thanks @PatrickXYS
/lgtm

[Bobgy]

/approve
I don't have any particular objections, I'm not very familiar with kf testing infra

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Bobgy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[PatrickXYS]

False-positive signal for presubmit, it actually succeed.

http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-testing-presubmit-py-unittests-755-0c30576-5072-7695?tab=workflow

[PatrickXYS]

/test kubeflow-testing-presubmit