Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup PR testing for build-pipeline integration tests #16

Closed
bobcatfish opened this issue Sep 7, 2018 · 0 comments
Closed

Setup PR testing for build-pipeline integration tests #16

bobcatfish opened this issue Sep 7, 2018 · 0 comments
Assignees
@bobcatfish
Milestone
Mid October Demo

Comments

@bobcatfish
Copy link
Collaborator

Need to complete #15 before we can tackle this

Expected Behavior

Every PR should have integration tests run against it automatically.

(The long term dream is that we use the Pipeline CRD reference implementation itself to test this repo, but since it is just an API right now, that's a ways off!)

Actual Behavior

Even once #15 is done, we won't have anything to ensure we can actually deploy the CRDs and use them with a running controller.

Steps to Reproduce the Problem

To run the current equivalent of "integration tests", manually walk through the "installing and running" steps in the DEVELOPMENT.md:

# Add/update CRDs in your kubernetes cluster
make install

# Run your controller locally, (stop execution with `ctrl-c`)
make run

# In another terminal, deploy tasks
kubectl apply -f config/samples

Additional Info

We'll need a k8s cluster to deploy to, probably we can make use of boskos:

Whatever we implement should be easy for contributors to trigger manually and not require specific credentials.
This was referenced Sep 10, 2018
Merged
Merged
@bobcatfish bobcatfish mentioned this issue Sep 17, 2018
Closed
@bobcatfish bobcatfish changed the title Setup CI for build-pipeline integration tests Setup PR testing for build-pipeline integration tests Sep 18, 2018
@bobcatfish bobcatfish mentioned this issue Sep 21, 2018
Closed
@bobcatfish bobcatfish added this to the Mid October Demo milestone Sep 21, 2018
This was referenced Sep 21, 2018
Closed
Closed
Closed
Closed
Closed
@bobcatfish bobcatfish self-assigned this Sep 27, 2018
bobcatfish added a commit to bobcatfish/serving that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
6b28f68 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

I also fixed some headings which were duplicated.
@bobcatfish bobcatfish mentioned this issue Sep 27, 2018
Merged
bobcatfish added a commit to bobcatfish/pkg that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
9c14dd6 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.
bobcatfish added a commit to bobcatfish/pkg that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
8cc6a87 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.
bobcatfish added a commit to bobcatfish/pkg that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
838ad6b 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.
bobcatfish added a commit to bobcatfish/pkg that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
0edefaa 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.
bobcatfish added a commit to bobcatfish/pkg that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
a7ab2e1 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.
@bobcatfish bobcatfish mentioned this issue Sep 27, 2018
Merged
bobcatfish added a commit to bobcatfish/serving that referenced this issue Sep 27, 2018
@bobcatfish
Move docs from knative/serving to knative/pkg
ee3a050 
While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

I also fixed some headings which were duplicated.

Adding docs to knative/pkg in knative/pkg#102
bobcatfish referenced this issue in bobcatfish/pipeline Sep 28, 2018
@bobcatfish
Add skeleton integration test
b265ef8 
In this first step, we add an empty integration test that does nothing
but list `Pipeline`s in a namespace harded coded to `arendelle`
(already used the Toy Story theme in knative/serving, needed something
different here, thanks @aaron-prindle XD).

This can be run manually against a developer's own cluster, so folks can
start adding integration tests in parallel if they want to.

This is based off the end to end tests in knative/serving
(https://github.com/knative/serving/tree/master/test#running-end-to-end-tests)
and uses the libs in knative/pkg extensively, which were made from the
libs in knative/serving. The docs weren't moved with the libs tho which
I'm trying to fix in:
* knative/serving#2097
* knative/pkg#102

This is the first step for #16, next I'll be looking into executing these
via Prow, either against a hardcoded cluster with the secrets in Prow or
going straight for kubetest + boskos.
bobcatfish referenced this issue in bobcatfish/pipeline Sep 28, 2018
@bobcatfish
Add skeleton integration test
9b3b398 
In this first step, we add an empty integration test that does nothing
but list `Pipeline`s in a namespace harded coded to `arendelle`
(already used the Toy Story theme in knative/serving, needed something
different here, thanks @aaron-prindle XD).

This can be run manually against a developer's own cluster, so folks can
start adding integration tests in parallel if they want to.

This is based off the end to end tests in knative/serving
(https://github.com/knative/serving/tree/master/test#running-end-to-end-tests)
and uses the libs in knative/pkg extensively, which were made from the
libs in knative/serving. The docs weren't moved with the libs tho which
I'm trying to fix in:
* knative/serving#2097
* knative/pkg#102

The commit adds opencensus, since this is used by the test libs for
collecting and emitting metrics. We might want to use those eventually
too so that seems fine tho it would be nice if they were optional
¯\_(ツ)_/¯

This is the first step for #16, next I'll be looking into executing these
via Prow, either against a hardcoded cluster with the secrets in Prow or
going straight for kubetest + boskos.
@bobcatfish bobcatfish mentioned this issue Sep 28, 2018
Merged
bobcatfish referenced this issue in bobcatfish/pipeline Oct 2, 2018
@bobcatfish
Add skeleton integration test
7db3ef6 
In this first step, we add an empty integration test that does nothing
but list `Pipeline`s in a namespace harded coded to `arendelle`
(already used the Toy Story theme in knative/serving, needed something
different here, thanks @aaron-prindle XD).

This can be run manually against a developer's own cluster, so folks can
start adding integration tests in parallel if they want to.

This is based off the end to end tests in knative/serving
(https://github.com/knative/serving/tree/master/test#running-end-to-end-tests)
and uses the libs in knative/pkg extensively, which were made from the
libs in knative/serving. The docs weren't moved with the libs tho which
I'm trying to fix in:
* knative/serving#2097
* knative/pkg#102

The commit adds opencensus, since this is used by the test libs for
collecting and emitting metrics. We might want to use those eventually
too so that seems fine tho it would be nice if they were optional
¯\_(ツ)_/¯

This is the first step for #16, next I'll be looking into executing these
via Prow, either against a hardcoded cluster with the secrets in Prow or
going straight for kubetest + boskos.
bobcatfish referenced this issue in bobcatfish/pipeline Oct 2, 2018
@bobcatfish
Add skeleton integration test
272b618 
In this first step, we add an empty integration test that does nothing
but list `Pipeline`s in a namespace harded coded to `arendelle`
(already used the Toy Story theme in knative/serving, needed something
different here, thanks @aaron-prindle XD).

This can be run manually against a developer's own cluster, so folks can
start adding integration tests in parallel if they want to.

This is based off the end to end tests in knative/serving
(https://github.com/knative/serving/tree/master/test#running-end-to-end-tests)
and uses the libs in knative/pkg extensively, which were made from the
libs in knative/serving. The docs weren't moved with the libs tho which
I'm trying to fix in:
* knative/serving#2097
* knative/pkg#102

The commit adds opencensus, since this is used by the test libs for
collecting and emitting metrics. We might want to use those eventually
too so that seems fine tho it would be nice if they were optional
¯\_(ツ)_/¯

This is the first step for #16, next I'll be looking into executing these
via Prow, either against a hardcoded cluster with the secrets in Prow or
going straight for kubetest + boskos.
knative-prow-robot pushed a commit to knative/pkg that referenced this issue Oct 2, 2018
@bobcatfish @knative-prow-robot
Move docs from knative/serving to knative/pkg (#102)
49f84a6 
* Move docs from knative/serving to knative/pkg

While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

Removing docs from knative/serving in knative/serving#2097.

* Fix knative/serving specific docs

Fixed copy-pasted references to `knative/serving/test` which should be
`knative/pkg/test`.

Fixed examples which were specific to `knative/serving`; in the case of
the command arg examples, removed the parts which are specific to those
e2e tests (though this would make them harder to use by copy and paste,
but since the libs are in a common repo now anyway that's probably
okay).

Tried to make the metric legend easier to read.
knative-prow-robot pushed a commit that referenced this issue Oct 2, 2018
@bobcatfish @knative-prow-robot
Add skeleton integration test (#74)
fbad053 
* Add skeleton integration test

In this first step, we add an empty integration test that does nothing
but list `Pipeline`s in a namespace harded coded to `arendelle`
(already used the Toy Story theme in knative/serving, needed something
different here, thanks @aaron-prindle XD).

This can be run manually against a developer's own cluster, so folks can
start adding integration tests in parallel if they want to.

This is based off the end to end tests in knative/serving
(https://github.com/knative/serving/tree/master/test#running-end-to-end-tests)
and uses the libs in knative/pkg extensively, which were made from the
libs in knative/serving. The docs weren't moved with the libs tho which
I'm trying to fix in:
* knative/serving#2097
* knative/pkg#102

The commit adds opencensus, since this is used by the test libs for
collecting and emitting metrics. We might want to use those eventually
too so that seems fine tho it would be nice if they were optional
¯\_(ツ)_/¯

This is the first step for #16, next I'll be looking into executing these
via Prow, either against a hardcoded cluster with the secrets in Prow or
going straight for kubetest + boskos.

* Create namespace for tests to use

Before running any tests, create a randomized namespace, and tear down
the namespace at the end of the test. Because we're using `os.Exit`, we
can't use `defer` to execute the cleanup so we'll just do it the old
fashioned way.

Also added an empty `tearDown` to the integration test to demonstrate
how it should be added (specifically to make sure to run it on
interrupt as well).

* Make `namespace` private to `test` package

I made it public (`Namespace`) because of briefly misunderstanding the
scope of private variables (I thought they were limited to the file, but
they are available to the whole package).

Also changed some methods to use the global variable, this seems
generally worse but on the other hand there's no reason at the moment
for these methods to operate on any namespace other than the global one.
bobcatfish added a commit to bobcatfish/test-infra that referenced this issue Oct 2, 2018
@bobcatfish
Add trigger for integration tests
52ec47c 
The integration tests entrypoint in `presubmit-tests.sh`
(https://github.com/knative/build-pipeline/blob/master/test/presubmit-tests.sh#L46)
currently will just echo a `TODO` but I'd like to add this to Prow
before adding the logic to `presubmit-tests.sh` (especially b/c I don't
think I'll have permission to manually run these tests against the
boskos clusters in
https://github.com/knative/test-infra/blob/master/ci/prow/boskos/resources.yaml).

This is part of tektoncd/pipeline#16
@bobcatfish bobcatfish mentioned this issue Oct 2, 2018
Merged
knative-prow-robot pushed a commit to knative/test-infra that referenced this issue Oct 2, 2018
@bobcatfish @knative-prow-robot
Add trigger for integration tests (#167)
b29ff75 
The integration tests entrypoint in `presubmit-tests.sh`
(https://github.com/knative/build-pipeline/blob/master/test/presubmit-tests.sh#L46)
currently will just echo a `TODO` but I'd like to add this to Prow
before adding the logic to `presubmit-tests.sh` (especially b/c I don't
think I'll have permission to manually run these tests against the
boskos clusters in
https://github.com/knative/test-infra/blob/master/ci/prow/boskos/resources.yaml).

This is part of tektoncd/pipeline#16
bobcatfish added a commit to bobcatfish/test-infra that referenced this issue Oct 3, 2018
@bobcatfish
Allow base name of resource to be longer
c0be240 
When trying to run this script for knative/build-pipeline, the resource
name was getting truncated too early: 1) the resource names were all the
same and 2) they were invalid because they happened to end with `-`.

This resulted in errors like:
```
ERROR: (gcloud.compute.networks.describe) Could not fetch resource:
 - Invalid value 'kbuild-pipeline-e2e-'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?'
```

30 chars is pretty arbitrary so open to better solutions also!

This is for tektoncd/pipeline#16
@bobcatfish bobcatfish mentioned this issue Oct 3, 2018
Closed
bobcatfish referenced this issue in bobcatfish/pipeline Oct 3, 2018
@bobcatfish
Run integration tests on presubmit using boskos
2de59f6 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (I assume using
boskos?). I'll follow up on this more later on and update the docs when
I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
bobcatfish referenced this issue in bobcatfish/pipeline Oct 3, 2018
@bobcatfish
Run integration tests on presubmit using boskos (I think?)
c01b201 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (I assume using
boskos?). I'll follow up on this more later on and update the docs when
I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
bobcatfish referenced this issue in bobcatfish/pipeline Oct 3, 2018
@bobcatfish
Run integration tests on presubmit using boskos (I think?)
88430bb 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (I assume using
boskos?). I'll follow up on this more later on and update the docs when
I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
@bobcatfish bobcatfish mentioned this issue Oct 3, 2018
Merged
bobcatfish referenced this issue in bobcatfish/pipeline Oct 3, 2018
@bobcatfish
Run integration tests on presubmit using boskos (I think?)
863ce6c 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (I assume using
boskos?). I'll follow up on this more later on and update the docs when
I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
bobcatfish referenced this issue in bobcatfish/pipeline Oct 3, 2018
@bobcatfish
Run integration tests on presubmit
3061d00 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (Using boskos if
running from Prow, or kubetest otherwise). I'll follow up on this more
later on and update the docs when I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
knative-prow-robot pushed a commit to knative/serving that referenced this issue Oct 4, 2018
@bobcatfish @knative-prow-robot
Move docs from knative/serving to knative/pkg (#2097)
78eba97 
* Move docs from knative/serving to knative/pkg

While working on tektoncd/pipeline#16
I started trying to build some integration tests, and I wanted to use
the code which had been moved to knative/pkg, but it turned out the docs
hadn't been ported with the code, so I am moving the docs relevant to
knative/pkg/test to that repo.

I also fixed some headings which were duplicated.

Adding docs to knative/pkg in knative/pkg#102

* Fix name of function in docs

lol this one was wrong THE WHOLE TIME SINCE THE VERY FIRST COMMIT where
I used one name in the implementation and a different name in the docs
XD 5bc7377
bobcatfish referenced this issue in bobcatfish/pipeline Oct 4, 2018
@bobcatfish
Run integration tests on presubmit
7b16748 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (Using boskos if
running from Prow, or kubetest otherwise). I'll follow up on this more
later on and update the docs when I have more info.

Assuming we can get knative/test-infra#169
merged I'll revert the change to `e2e-tests.sh` in the `vendor` dir.

Fixes #16
bobcatfish referenced this issue in bobcatfish/pipeline Oct 4, 2018
@bobcatfish
Run integration tests on presubmit
1b3fc26 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (Using boskos if
running from Prow, or kubetest otherwise). I'll follow up on this more
later on and update the docs when I have more info.

Fixes #16
knative-prow-robot pushed a commit that referenced this issue Oct 4, 2018
@bobcatfish @knative-prow-robot
Run integration tests on presubmit
1d8f2b4 
Using the scripts from knative/test-infra, run the integration tests
against a cluster created with boskos. A lot of the boskos stuff is
still kind of magical, for example I have no idea what is controlling
the $PROJECT_ID that is being used by knative projects on presubmit, but
after wrestling with it a bit and hacking around `--run-tests` I was
able to get it to work locally and deploy a cluster (Using boskos if
running from Prow, or kubetest otherwise). I'll follow up on this more
later on and update the docs when I have more info.

Fixes #16
bobcatfish referenced this issue in bobcatfish/pipeline Oct 4, 2018
@bobcatfish
Add badge to README for integration tests
b80e183 
Integration tests were added as part of #16
@bobcatfish bobcatfish mentioned this issue Oct 4, 2018
Merged
knative-prow-robot pushed a commit that referenced this issue Oct 5, 2018
@bobcatfish @knative-prow-robot
Add badge to README for integration tests
29d25e5 
Integration tests were added as part of #16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bobcatfish bobcatfish

Labels
None yet
Projects
None yet
Milestone
Mid October Demo
Development

No branches or pull requests
1 participant
@bobcatfish