tests: Scripts for e2e tests

[kimwnasptd]

This is a first step for #2099

The PR reworks the entire tests repo to include a simple e2e test that:

1. Installs all the KF manifests
2. Runs the E2E Notebooks we have in https://github.com/kubeflow/pipelines/blob/master/samples/contrib/kubeflow-e2e-mnist/kubeflow-e2e-mnist.ipynb

While developing the tests locally I was using KinD to spin up clusters and test everything from scratch. With this mentality, and along the fact that we don't have many maintainers in the kubeflow/testing repo, I tried to use GH actions to run these E2E tests.

But the performance was abysmal. Even with optimizations, like loading etcd in memory kubernetes-sigs/kind#845, I couldn't get the tests to succeed. It took more than 45 mins to apply all the manifests and then the test failed due to etcd timeouts.

I've included the GH workflow files, to give it one last try, but I don't have many expectations. The next step for this is to see how could we use the AWS infra/prow to spin up the clusters and try our luck with there. The current blocking PR for this is kubeflow/testing#972.

cc @kubeflow/release-team
/cc @StefanoFioravanzo @elikatsis

[kimwnasptd]

The PR also broke the unit tests. I'll send a fix for them. We should be able to run unit tests for GH actions though, since this is not resource intensive.

/assign @kimwnasptd

[kimwnasptd]

After inspecting the VM sizes of the GH actions I realized that the machines were running out of memory when I was installing all the manifests. It's a 7Gb VM.

I managed to created a "light" version of the manifests that only install the barebone components, to get the tests running. This time it took 6mins to install, but in the end got timeouts during training. This is most probably due to CPU limitations.

@StefanoFioravanzo @elikatsis I'm going to remove the GH action and fix unit tests. Then as a next step I'll implement this by using Prow.

[kimwnasptd]

@elikatsis @StefanoFioravanzo this should be ready for review. To summarize:

1. I've removed old the deprecated files in the tests dir
2. I've updated the unit tests to build the example kustomization, as a quick sanity check that most of the components's files can be generated
3. I've prepared some helpers in hack for spinning up a KinD cluster locally, so that someone can test the e2e scripts
4. I've created tests/e2e scripts, which we can later use with Prow

The next step will be to use Prow, with workflows similar to what we do for Notebooks, in order to run our e2e tests there. https://github.com/kubeflow/kubeflow/blob/master/prow_config.yaml

[elikatsis]

Thanks for all the effort and the troubleshooting @kimwnasptd. Orchestrating all of this testing.. phew.. That's great!

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: elikatsis, kimwnasptd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [elikatsis,kimwnasptd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment