upgrade ks app for E2E test to 0.3.0

[stpabhi]

This PR updates api version to 0.3.0 so that test infra ksonnet version is upgraded to v0.13.1 as we are recommending the same in the docs.

Signed-off-by: Abhilash Pallerlamudi stp.abhi@gmail.com

---

This change is 

[stpabhi]

/assign @jlewi
/assign @r2d4

[stpabhi]

/test kubeflow-presubmit

[jlewi]

It looks like there was a problem starting the tests.

You can follow these instructions
https://github.com/kubeflow/testing#no-results-show-up-in-gubernator

To use the prow jobs UI to get pod logs to see what went wrong.

[stpabhi]

/test kubeflow-presubmit

[stpabhi]

@jlewi tried to get pod logs but not enough information. I tried to setup a local kubeflow-test-infra and ran a debug pod to check and found that ks-13 binary is missing in /usr/local/bin. Do you have access to debug pod on prow?

Update: found that ks-13 tarball was not downloaded. Will check the Dockerfile and open a fix.

[stpabhi]

gcr.io/kubeflow-ci/test-worker:latest is old and doesn't have the ks-13 binaries. Should we update the tag?

[jlewi]

@stpabhi You can update the image. You should pin to a specific image like we do here
https://github.com/kubeflow/examples/blob/89e960202abf9b065c67bf96b99894564fdb7f37/test/workflows/components/mnist.jsonnet#L18

As opposed to using a relative tag.

Did you run ks upgrade to upgrade the app?

The reason I ask is because I would have thought upgrading to 0.13 made other changes such as the location of .libsonnet files.

[stpabhi]

Yes, I did run ks upgrade. It complains that upgraded lib already exists which is go/src/github.com/kubeflow/kubeflow/testing/workflows/lib/ksonnet-lib/v1.7.0. I will update the testing images to one that has ks-13 bin.

[stpabhi]

@jlewi Image has to be updated and merged before updating app?

[jlewi]

@stpabhi It shouldn't. The presubmits will use whatever workflow you have defined on the PR. So if your PR updates the image used for the worker then that's what should get used.

[stpabhi]

@jlewi Got it. I’m not sure what else I have missed. Is it possible to run on Minikube without gcloud services so that it’s easy to debug?

[jlewi]

Your test failed.

Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 386, in <module> final_result = main() File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 376, in main return run(args, file_handler) File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 168, in run util.run([ks_cmd, "version"]) File "/src/kubeflow/testing/py/kubeflow/testing/util.py", line 58, in run command, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child raise child_exceptionOSError: [Errno 2] No such file or directory

I think its refering to binary ks-13 not being found.

I think your image is too old. try using
gcr.io/kubeflow-ci/test-worker/test-worker:v20190116-b7abb8d-e3b0c4

[stpabhi]

@jlewi I did use the new images. Let me try again using tags instead of sha checksum

[jlewi]

I think the sha you were using was pointing to a different image from the one I pasted. The one I pasted should be used by our mnist example which is why I have confidence its up to date.

[jlewi]

Still seems like its not running. I think to debug it you should submit the test; and then use the prow jobs dashboard to fetch pod logs to see what's failing.

I'm going to mark this WIP while you work on fixing the test; when its fixed please remove WIP from the title.

If you need help lets chat on kubeflow.slack.com

[stpabhi]

/test kubeflow-presubmit

[stpabhi]

@jlewi App version 0.2.0 worked. I tried on my gke instance with ks-13 but ran into another issue.

[stpabhi]

@jlewi Weird, App version 0.3.0 works now. Could you please review and add the appropriate labels?

[jlewi]

Reviewed 14 of 14 files at r1.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @abhi-g and @ellis-bigelow)

[jlewi]

/lgtm
/approve

[stpabhi]

/test kubeflow-presubmit

[jlewi]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stpabhi]

/test kubeflow-presubmit

[lluunn]

Why do we change back to old test-worker image?

[lluunn]

It breaks to deployapp test since it needs the new change in test worker

[stpabhi]

@lluunn Is this specific to #2463? Notebooks test did not fail in the presubmit test for this PR https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/2338/kubeflow-presubmit/5676/

[lluunn]

no, it's periodic test.
https://k8s-testgrid.appspot.com/sig-big-data#kubeflow-periodic-master