[Test] Post submit tests on MKP deployment fails

[rui5i]

What happened:
Post submit tests on MKP deployment fails. Due to docker command can not get run.
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/kubeflow-pipeline-postsubmit-mkp-component-test/1223274544043659266

What did you expect to happen:
Docker command cannot run on base image.
log:
sudo docker pull gcr.io/cloud-marketplace-tools/k8s/dev
903
/home/prow/go/src/github.com/kubeflow/pipelines/test/deploy-pipeline-mkp-cli.sh: line 45: sudo: command not found

Anything else you would like to add:
There is no sudo in the base image when installing docker:
https://github.com/kubernetes/test-infra/blob/master/images/bootstrap/Dockerfile#L94
One thing we could try is to add group(https://docs.docker.com/install/linux/linux-postinstall/) without sudo on the base image.
Or remove all the docker build and replace with cloud build.

[Bobgy]

QQ: did you try removing the sudo? I think docker runs as root user by default. Not very sure about this, if you haven't tried, then it may work

[rui5i]

Yes. First version is without sudo. And we found that docker requires sudo by default. To avoid running with sudo, we need set up docker post installation(https://docs.docker.com/install/linux/linux-postinstall/).

[Bobgy]

SG
FYI, https://hub.docker.com/_/docker/
Docker has official dind images. If patching existing image doesn't work. Backup plan can be moving to a separate pod running dind image.

[rmgogogo]

It's docker insider docker case? We can

1. move the "docker tag" to cloud build
   https://github.com/kubeflow/pipelines/blob/master/.cloudbuild.yaml

2. use another baseimage which already have mpdev

[rui5i]

Besides mpdev, we also need docker build for deployer image(https://github.com/kubeflow/pipelines/blob/master/test/deploy-pipeline-mkp-cli.sh#L64). An alternative way to do this is to build the deployer image using Cloud Build. But in that case the behavior we are testing is not the same as our release process. Can we start with install sudo to see if it works or not?

[numerology]

Mirror the offline discussion here:

Another alternative is to maintain our own test image used in prow (other than kubekins-e2e image built by k8s). Note that this is the practice of other Kubeflow repo, which is currently using gcr.io/kubeflow-ci/test-worker image

[rui5i]

Post submit tests are fixed in this PR: #3193.