Modify presubmits to support testing with v1alpha2

[jlewi]

Changes to support v1alpha2 testing in presubmits.

• The tests are currently disabled because they aren't passing yet because
  termination policy isn't handled correctly (TFJob not marked as success when master exits but not workers #634)

• Changed the v1alpha2 test to use the same smoke test as used by v1alpha1 as
  opposed to using mnist.
  mnist causing problems because of issues downloading the data
  see Presubmit failures; Timeout waiting for TFJob v1alpha2 job kubeflow#974

• We want a simpler test that allows for more direct testing of the distributed
  communication pattern

• Also mnist is expensive in that it tries to download data.

• Add a parameter tfJobVersion to the deploy script so we can control
  whether we deploy v1alpha1 or v1alpha2

• Parameterize the E2E test workflow by the TFJob version we want to run.

• update test-app - We need to pull in a version of the app which
  has the TFJobVersion flag.

• Create a script to regenerate the test-app for future use.

Related to #589

---

This change is 

[coveralls]

Coverage decreased (-0.9%) to 55.067% when pulling 3463fcc on jlewi:fix_tfjob into e164ba5 on kubeflow:master.

[jlewi]

pylint failures should be fixed by kubeflow/testing#156

The other error looks like a problem with ks being too old in the test image after we upgraded the test app

This should be fixed by kubeflow/testing#155

[jlewi]

/test all

[jlewi]

/test all

[jlewi]

Most recent failure:

 wait_for_deployment
   name, namespace))
kubeflow.testing.util.TimeoutError: Timeout waiting for deployment tf-job-operator in namespace kubeflow

[jlewi]

Looking at the event logs I see the error

{
 insertId:  "1t7pnmeg2uox2aj"  
 jsonPayload: {
  apiVersion:  "v1"   
  involvedObject: {…}   
  kind:  "Event"   
  message:  "Failed to apply default image tag "gcr.io/kubeflow-ci/tf_operator:": couldn't parse image reference "gcr.io/kubeflow-ci/tf_operator:": invalid reference format"   
  metadata: {…}   
  reason:  "InspectFailed"   
  source: {…}   
  type:  "Warning"   
 }
 logName:  "projects/kubeflow-ci/logs/events"  
 receiveTimestamp:  "2018-06-12T20:02:03.841513544Z"  
 resource: {…}  
 severity:  "WARNING"  
 timestamp:  "2018-06-12T20:01:59Z"  
}

[jlewi]

Exec into debug worker and check the ksonnet test app

ks param list --env=e2e-0612-1959-3cf6
COMPONENT PARAM                   VALUE
========= =====                   =====
core      cloud                   "null"
core      disks                   "null"
core      jupyterHubAuthenticator "null"
core      jupyterHubImage         "gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1"
core      jupyterHubServiceType   "ClusterIP"
core      jupyterNotebookPVCMount "null"
core      jupyterNotebookRegistry "gcr.io"
core      jupyterNotebookRepoName "kubeflow-images-public"
core      name                    "kubeflow-core"
core      namespace               "kubeflow"
core      reportUsage             "false"
core      tfAmbassadorImage       "quay.io/datawire/ambassador:0.30.1"
core      tfAmbassadorServiceType "ClusterIP"
core      tfDefaultImage          "null"
core      tfJobImage              "gcr.io/kubeflow-ci/tf_operator:"
core      tfJobUiServiceType      "ClusterIP"
core      tfJobVersion            "v1alpha1"
core      tfStatsdImage           "quay.io/datawire/statsd:0.30.1"
core      usageId                 "unknown_cluster"

So looks like tfJobImage wasn't set correctly

Argo logs look like the image isn't set correctly

INFO:root:Running: ks param set --env=e2e-0612-1959-3cf6 core tfJobImage gcr.io/kubeflow-ci/tf_operator:
cwd=/mnt/test-data-volume/kubeflow-tf-operator-presubmit-tfjob-e2e-632-54931b6-595-0276/src/kubeflow/tf-operator/test/test-app

[jlewi]

My suspicion is that when we pushed a new testing worker image; we pushed some updates to the run_e2e_workflow.py and that broke things.

[jlewi]

If params.versionTag in the workflow isn't set we should use the name
https://github.com/kubeflow/tf-operator/blob/e164ba54c1ffdd750f059711beb65c9a5936c684/test/workflows/components/workflows.libsonnet#L62

I changed versionTag from null to "". That's the problem. I made this change because ks 0.11 was having some problems with null.

[jlewi]

Tests are passing; this is ready for review.

/assign @ankushagarwal
/assign @gaocegege

[ankushagarwal]

/lgtm
/approve

[gaocegege]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ankushagarwal, gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment