Integrate with Prow for Continuous Testing

[jlewi]

We would like to integrate with Prow, K8s, test infra to provide continuous testing.

After some initial discussion with @foxish and @krzyzacy the current plan is to have 2 prow jobs;

• First job will checkout an build a container containing the CRD and E2E tests.
• Second job will install the CRD and run the E2E tests.

The current plan is to not rely on helm test but just to write go tests as necessary.

First step is probably to create a prow job to build and push the Docker image.

Remaining Issues Blocking This Issue

• Post submit jobs don't correctly upload artifacts to GCS #62 Post submit jobs not uploaded correctly
• No results show up in prow test grid for presubmit jobs #75 Presubmit results not showing up in test grid
• No results show up if you click on mlkube-build-periodic #76 No results show up for periodic jobs
• runner.py needs to create build-log.txt with stdout/stderr of test #82 Need to copy stdout/stderr of tests to GCS.
• Update prow to use repo tensorflow/k8s #84 Update prow to use tensorflow/k8s
• postsubmit results aren't showing up in testrgrid #113 postsubmit results show up in test grid

[jlewi]

Initial ProwJobs were defined in kubernetes/test-infra/pull/4951.

Next steps

• Configure mlkube.io to send webhooks to Prow so that we trigger Pre/Postsubmits.
• Actually bring up a GKE cluster and deploy the CRD as part of the test.

[jlewi]

Status

• A complete E2E test was submitted in E2E test for the CRD #49
• Prow is running pre & post submit prow jobs
• Periodic jobs don't appear to be running
• No results are showing up in the Test Grid
• K8s chatbot can't update test status on PRs; waiting on owner of chatbot account to accept invite to be an admin of the mlkube.io repository
• Need to add repository to the PR dashboard instructions

[krzyzacy]

the periodic job was keep failing without logs - http://prow.k8s.io/?type=periodic&job=mlkube-build-periodic, I'll retrigger and see what is going on

pr/postsubmit jobs probably need a different log path in testgrid

[krzyzacy]

INFO:root:Image info:
{
  "image": "gcr.io/mlkube-testing/builder:v20171017-98ba4b8-dirty"
}
INFO:root:Environment variables REPO_OWNER and REPO_NAME not set; not checking out code.
Traceback (most recent call last):
  File "/workspace/bootstrap.py", line 118, in <module>
    main()
  File "/workspace/bootstrap.py", line 110, in main
    src_dir, sha = clone_repo()
  File "/workspace/bootstrap.py", line 64, in clone_repo
    raise ValueError("No code found at %s", dest)
ValueError: ('No code found at %s', '/go/src/github.com/jlewi/mlkube.io')

[jlewi]

@krzyzacy ; thanks. I'll put together a fix for the periodic issue.

[jlewi]

#56 hopefully fixes the periodic test; should run every 8 hours so should know tomorrow morning.

[jlewi]

@krzyzacy My fix for the periodic jobs doesn't appear to have worked. Could you check the logs or tell me how I can check them?

[krzyzacy]

the job comes from http://prow.k8s.io/?job=mlkube-build-periodic, but since it failed early on, it did not upload any logs :-( and the failing pod will be deleted within next hour.. You can try click rerun and run the kubectl command it pops up. (I'll retrigger and check the log)

[jlewi]

@krzyzacy Thanks; I was looking at the wrong tab; http://prow.k8s.io/?repo=jlewi%2Fmlkube.io. I realize now periodic jobs don't specify the repository so they wouldn't show up under the repository tab.

Here's the error; another bug on my end.

Traceback (most recent call last):
  File "/workspace/bootstrap.py", line 110, in <module>
    main()
  File "/workspace/bootstrap.py", line 106, in main
    run(["python", runner, "--src_dir=" + src_dir, "--sha=" + sha])
TypeError: cannot concatenate 'str' and 'NoneType' objects

[jlewi]

k8s-bot integration is now working; see #50

[jlewi]

Filed #59 to fix an issue with presubmits.

[jlewi]

#59 is fixed

Periodic jobs are fixed. http://prow.k8s.io/?job=mlkube-build-periodic

[jlewi]

Results are showing up in testgrid for postsubmit and periodic jobs but not presubmit jobs.

Filed #75

[jlewi]

All blocking issues have been fixed. Prow integration is all setup.