Run integration tests on presubmit using boskos (I think?)

Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (I assume using
boskos?). I'll follow up on this more later on and update the docs when
I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16

test/README.md

@@ -127,4 +127,29 @@ You can run this locally with:
 
 ```shell
 test/presubmit-tests.sh
-```
+test/presubmit-tests.sh --build-tests
+test/presubmit-tests.sh --unit-tests
+```
+
+The configuration for these tests lives
+[in the `knative/test-infra` Prow configuration](https://github.com/knative/test-infra/blob/master/ci/prow/config.yaml).
+
+### Running presubmit integration tests
+
+By default the integration tests will try to get a new cluster using [boskos](https://github.com/kubernetes/test-infra/tree/master/boskos) and
+[these hardcoded GKE projects](https://github.com/knative/test-infra/blob/master/ci/prow/boskos/resources.yaml#L15),
+which only [the `knative/test-infra` OWNERS](https://github.com/knative/test-infra/blob/master/OWNERS)
+have access to.
+
+If you would like to run the integration tests against your cluster, you can use the
+`K8S_CLUSTER_OVERRIDE` environment variable to force the scripts to use your own cluster,
+and provide `KO_DOCKER_REPO` (as specified in the [DEVELOPMENT.md](../DEVELOPMENT.md#environment-setup)):
+
+```shell
+export K8S_CLUSTER_OVERRIDE=my_k8s_cluster # corresponds to a `context` in your kubeconfig
+export KO_DOCKER_REPO=gcr.io/my_docker_repo # required for deployments using `ko`
+test/presubmit-tests.sh --integration-tests
+```
+
+Otherwise, the script will attempt to use [boskos](https://github.com/kubernetes/test-infra/tree/master/boskos)
+to create a new cluster in the script `$PROJECT_ID`.

test/cluster-setup.sh

@@ -0,0 +1,75 @@
+#!/bin/bash
+
+# Copyright 2018 The Knative Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script calls out to scripts in knative/test-infra to setup a cluster
+# and deploy the Pipeline CRD to it for running integration tests.
+
+source $(dirname $0)/../vendor/github.com/knative/test-infra/scripts/e2e-tests.sh
+
+set -o xtrace
+set -o errexit
+set -o pipefail
+
+function take_down_pipeline() {
+    header "Tearing down Pipeline CRD"
+    ko delete --ignore-not-found=true -f config/
+}
+trap take_down_pipeline EXIT
+
+# Called by `fail_test` (provided by `e2e-tests.sh`) to dump info on test failure
+function dump_extra_cluster_state() {
+  for crd in pipelines pipelineruns tasks taskruns resources pipelineparams
+  do
+    echo ">>> $crd:"
+    kubectl get $crd -o yaml --all-namespaces
+  done
+  echo ">>> Pipeline controller log:"
+  kubectl -n knative-build-pipeline logs $(kubectl -n knative-build-pipeline get pods -l app=build-pipeline-controller -o name)
+  echo ">>> Pipeline webhook log:"
+  kubectl -n knative-build-pipeline logs $(kubectl -n knative-build-pipeline get pods -l app=build-pipeline-webhook -o name)
+}
+
+set +o xtrace
+header "Setting up environment"
+# The intialize method will attempt to create a new cluster in $PROJECT_ID unless
+# the `--run-tests` parameter is provided and `K8S_CLUSTER_OVERRIDE` is set, however
+# since knative/test-infra/scripts/presubmit-tests.sh doesn't propagate `--run-tests`
+# (and it's kind of a confusing param name), we'll infer it from the presence of
+# `K8S_CLUSTER_OVERRIDE`
+if [[ -z ${K8S_CLUSTER_OVERRIDE} ]]; then
+    initialize
+else
+    initialize --run-tests
+fi
+set -o xtrace
+
+# The scripts were initially setup to use the knative/serving style `DOCKER_REPO_OVERRIDE`
+# before knative/serving was updated to use `ko` + `KO_DOCKER_REPO`. If the scripts were
+# called with `KO_DOCKER_REPO` already set (i.e. using a user's own local cluster, we should
+# respect that).
+if ! [[ -z ${DOCKER_REPO_OVERRIDE} ]]; then
+    export KO_DOCKER_REPO=${DOCKER_REPO_OVERRIDE}
+fi
+
+# Deploy the latest version of the Pipeline CRD.
+# TODO(#59) do we need to deploy the Build CRD as well?
+header "Deploying Pipeline CRD"
+ko apply -f config/
+
+# Wait for pods to be running in the namespaces we are deploying to
+set +o xtrace
+wait_until_pods_running knative-build-pipeline || fail_test "Pipeline CRD did not come up"
+set -o xtrace

test/init_test.go

@@ -73,6 +73,7 @@ func tearDownMain(kubeClient *knativetest.KubeClient, logger *logging.BaseLogger
 func TestMain(m *testing.M) {
 	initializeLogsAndMetrics()
 	logger := logging.GetContextLogger("TestMain")
+	logger.Infof("Using kubeconfig at `%s` with cluster `%s`", knativetest.Flags.Kubeconfig, knativetest.Flags.Cluster)
 
 	namespace = AppendRandomString("arendelle")
 	kubeClient := createNamespace(namespace, logger)

test/presubmit-tests.sh

@@ -15,6 +15,9 @@
 # limitations under the License.
 
 # This script runs the presubmit tests; it is started by prow for each PR.
+# When running `--integration-tests`, use `K8S_CLUSTER_OVERRIDE` to point at your
+# own cluster (a context in your kubeconfig), or the script will attempt to invoke
+# boskos to obtain one of the knative testing clusters in GKE.
 
 set -o xtrace
 
@@ -43,7 +46,21 @@ function unit_tests() {
 
 function integration_tests() {
   echo "Running integration tests"
-  echo "TODO(#16): add integration tests"
+
+  # Use knative test-infra scripts to make `fail-test` and `success` available.
+  # which will output helpful information for debugging failures, and ensure test
+  # results are made available.
+  source $(dirname $0)/../vendor/github.com/knative/test-infra/scripts/e2e-tests.sh
+  # Setup the cluster (if not using an existing cluster) and deploy the Pipeline CRD
+  source $(dirname $0)/cluster-setup.sh
+
+  options=""
+  (( EMIT_METRICS )) && options="-emitmetrics"
+  report_go_test \
+    -v -tags=e2e -count=1 -timeout=20m ./test \
+    ${options} || fail_test
+
+  success
 }
 
 main $@