Skip to content
This repository has been archived by the owner on Jul 11, 2023. It is now read-only.

tests/e2e: introduce e2e framework basics, first test #1801

Merged
merged 15 commits into from
Oct 13, 2020
13 changes: 10 additions & 3 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
run:
tests: true

issues:
new: true

linters:
enable:
- goconst
Expand All @@ -19,3 +16,13 @@ linters:
linters-settings:
goimports:
local-prefixes: github.com/openservicemesh/osm

issues:
new: true
exclude-rules:
# Ignore error for ginkgo and gomega dot imports
- linters:
- golint
nojnhuh marked this conversation as resolved.
Show resolved Hide resolved
source: ". \"github.com/onsi/(ginkgo|gomega)\""
text: "dot imports"

4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ kind-up:
kind-reset:
kind delete cluster --name osm

.PHONY: test-e2e
test-e2e: docker-build-osm-controller docker-build-init build-osm
go test ./tests/e2e -test.v -ginkgo.v -ginkgo.progress -ctrRegistry $(CTR_REGISTRY) -osmImageTag $(CTR_TAG) -kindCluster

.env:
cp .env.example .env

Expand Down
10 changes: 10 additions & 0 deletions docs/development_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,16 @@ which tests the functionality of multiple functions together. In this particular
- [creates a ConfigMap](https://github.com/openservicemesh/osm/blob/release-v0.2/pkg/configurator/client_test.go#L32)
- [tests whether](https://github.com/openservicemesh/osm/blob/release-v0.2/pkg/configurator/client_test.go#L95-L96) the underlying functions compose correctly by fetching the results of the top-level function `GetMeshCIDRRanges()`

### End-to-End (e2e) Tests

End-to-end tests verify the behavior of the entire system. For OSM, e2e tests will install a control plane, install test workloads and SMI policies, and check that the workload is behaving as expected.

OSM's e2e tests are located in tests/e2e. The tests can be run using the `test-e2e` Makefile target. The Makefile target will also build the necessary container images and `osm` CLI binary before running the tests. The tests are written using Ginkgo and Gomega so they may also be directly invoked using `go test`. Be sure to build the `osm-controller` and `init` container images and `osm` CLI before directly invoking the tests. With either option, it is suggested to explicitly set the container registry location and tag to ensure up-to-date images are used by setting the `CTR_REGISTRY` and `CTR_TAG` environment variables.

In addition to the flags provided by `go test` and Ginkgo, there are several custom command line flags that may be used for e2e tests to configure global parameters like container image locations and cleanup behavior. The full list of custom flags can be found in [tests/e2e/common.go](/tests/e2e/common.go).

For more information, please refer to [OSM's E2E Readme](/tests/e2e/README.md).

#### Simulation / Demo
When we want to ensure that the entire system works correctly over time and
transitions state as expected - we run
Expand Down
5 changes: 4 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ require (
github.com/axw/gocov v1.0.0
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354 // indirect
github.com/deckarep/golang-set v1.7.1
github.com/docker/docker v1.4.2-0.20200203170920-46ec8731fbce
github.com/envoyproxy/go-control-plane v0.9.6
github.com/fatih/color v1.9.0
github.com/golang/mock v1.3.1
github.com/golang/protobuf v1.4.2
github.com/golangci/golangci-lint v1.30.0
Expand Down Expand Up @@ -39,10 +41,11 @@ require (
gopkg.in/yaml.v2 v2.3.0
helm.sh/helm/v3 v3.2.0
k8s.io/api v0.18.5
k8s.io/apimachinery v0.18.5
k8s.io/apimachinery v0.18.8
k8s.io/cli-runtime v0.18.5
k8s.io/client-go v0.18.5
rsc.io/letsencrypt v0.0.3 // indirect
sigs.k8s.io/kind v0.9.0
)

replace github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.2+incompatible
16 changes: 16 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ github.com/Venafi/vcert v0.0.0-20200310111556-eba67a23943f/go.mod h1:9EegQjmRoMq
github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alessio/shellescape v1.2.2 h1:8LnL+ncxhWT2TR00dfJRT25JWWrhkMZXneHVWnetDZg=
github.com/alessio/shellescape v1.2.2/go.mod h1:PZAiSCk0LJaZkiCSkPv8qIobYglO3FPpyFjDCtHLS30=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
Expand Down Expand Up @@ -243,9 +245,12 @@ github.com/envoyproxy/go-control-plane v0.9.6 h1:GgblEiDzxf5ajlAZY4aC8xp7DwkrGfa
github.com/envoyproxy/go-control-plane v0.9.6/go.mod h1:GFqM7v0B62MraO4PWRedIbhThr/Rf7ev6aHOOPXeaDA=
github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/evanphx/json-patch v0.0.0-20200808040245-162e5629780b/go.mod h1:NAJj0yf/KaRKURN6nyi7A9IZydMivZEm9oQLWNjfKDc=
github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/evanphx/json-patch v4.5.0+incompatible h1:ouOWdg56aJriqS0huScTkVXPC5IcNrDCXZ6OoTAWu7M=
github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/evanphx/json-patch/v5 v5.1.0 h1:B0aXl1o/1cP8NbviYiBMkcHBtUjIJ1/Ccg6b+SwCLQg=
github.com/evanphx/json-patch/v5 v5.1.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4=
github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d h1:105gxyaGwCFad8crR9dcMQWvV9Hvulu6hwUh4tWPJnM=
github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d/go.mod h1:ZZMPRZwes7CROmyNKgQzC3XPs6L/G2EJLHddWejkmf4=
github.com/fatih/camelcase v1.0.0/go.mod h1:yN2Sb0lFhZJUdVvtELVWefmrXpuZESvPmqwoZc+/fpc=
Expand Down Expand Up @@ -551,6 +556,7 @@ github.com/imdario/mergo v0.3.8 h1:CGgOkSJeqMRmt0D9XLWExdT4m4F1vd3FV3VPt+0VxkQ=
github.com/imdario/mergo v0.3.8/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jetstack/cert-manager v0.16.1 h1:ZmybpXT2g7wmZWzI765c+YZqjz+8BvmBVAoqm745gNM=
github.com/jetstack/cert-manager v0.16.1/go.mod h1:jLNsZnyuKeg5FkGWhI1H1eoikhsGEM1MpT5Z3Gh7oWk=
github.com/jingyugao/rowserrcheck v0.0.0-20191204022205-72ab7603b68a h1:GmsqmapfzSJkm28dhRoHz2tLRbJmqhU86IPgBtN3mmk=
Expand Down Expand Up @@ -745,6 +751,8 @@ github.com/pavel-v-chernykh/keystore-go v2.1.0+incompatible/go.mod h1:xlUlxe/2It
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pelletier/go-toml v1.8.0 h1:Keo9qb7iRJs2voHvunFtuuYFsbWeOBh8/P9v/kVMFtw=
github.com/pelletier/go-toml v1.8.0/go.mod h1:D6yutnOGMveHEPV7VQOuvI/gXY61bv+9bAOTRnLElKs=
github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/phayes/checkstyle v0.0.0-20170904204023-bfd46e6a821d h1:CdDQnGF8Nq9ocOS/xlSptM1N3BbrA6/kmaep5ggwaIA=
Expand Down Expand Up @@ -1101,6 +1109,8 @@ golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299 h1:DYfZAGf2WMFjMxbgTjaC+2HC7NkNAQs+6Q8b9WEB/F4=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed h1:J22ig1FUekjjkmZUM7pTKixYm8DvrYsvrBZdunYeIuQ=
golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20171227012246-e19ae1496984/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down Expand Up @@ -1260,6 +1270,8 @@ gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20190905181640-827449938966/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
helm.sh/helm/v3 v3.2.0 h1:V12EGAmr2DJ/fWrPo2fPdXWSIXvlXm51vGkQIXMeymE=
Expand All @@ -1284,6 +1296,8 @@ k8s.io/apimachinery v0.18.0 h1:fuPfYpk3cs1Okp/515pAf0dNhL66+8zk8RLbSX+EgAE=
k8s.io/apimachinery v0.18.0/go.mod h1:9SnR/e11v5IbyPCGbvJViimtJ0SwHG4nfZFjU77ftcA=
k8s.io/apimachinery v0.18.5 h1:Lh6tgsM9FMkC12K5T5QjRm7rDs6aQN5JHkA0JomULDM=
k8s.io/apimachinery v0.18.5/go.mod h1:OaXp26zu/5J7p0f92ASynJa1pZo06YlV9fG7BoWbCko=
k8s.io/apimachinery v0.18.8 h1:jimPrycCqgx2QPearX3to1JePz7wSbVLq+7PdBTTwQ0=
k8s.io/apimachinery v0.18.8/go.mod h1:6sQd+iHEqmOtALqOFjSWp2KZ9F0wlU/nWm0ZgsYWMig=
k8s.io/apiserver v0.18.0/go.mod h1:3S2O6FeBBd6XTo0njUrLxiqk8GNy6wWOftjhJcXYnjw=
k8s.io/apiserver v0.18.5/go.mod h1:+1XgOMq7YJ3OyqPNSJ54EveHwCoBWcJT9CaPycYI5ps=
k8s.io/cli-runtime v0.18.0 h1:jG8XpSqQ5TrV0N+EZ3PFz6+gqlCk71dkggWCCq9Mq34=
Expand Down Expand Up @@ -1333,6 +1347,8 @@ rsc.io/letsencrypt v0.0.3/go.mod h1:buyQKZ6IXrRnB7TdkHP0RyEybLx18HHyOSoTyoOLqNY=
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.7/go.mod h1:PHgbrJT7lCHcxMU+mDHEm+nx46H4zuuHZkDP6icnhu0=
sigs.k8s.io/controller-runtime v0.5.1-0.20200416234307-5377effd4043/go.mod h1:j4echH3Y/UPHRpXS65rxGXujda8iWOheMQvDh1uNgaY=
sigs.k8s.io/controller-tools v0.2.9-0.20200414181213-645d44dca7c0/go.mod h1:YKE/iHvcKITCljdnlqHYe+kAt7ZldvtAwUzQff0k1T0=
sigs.k8s.io/kind v0.9.0 h1:SoDlXq6pEc7dGagHULNRCCBYrLH6xOi7lqXTRXeAlg4=
sigs.k8s.io/kind v0.9.0/go.mod h1:cxKQWwmbtRDzQ+RNKnR6gZG6fjbeTtItp5cGf+ww+1Y=
sigs.k8s.io/kustomize v2.0.3+incompatible h1:JUufWFNlI44MdtnjUqVnvh29rR37PQFzPbLXqhyOyX0=
sigs.k8s.io/kustomize v2.0.3+incompatible/go.mod h1:MkjgH3RdOWrievjo6c9T245dYlB5QeXV4WCbnt/PEpU=
sigs.k8s.io/structured-merge-diff/v3 v3.0.0-20200116222232-67a7b8c61874/go.mod h1:PlARxl6Hbt/+BC80dRLi1qAmnMqwqDg62YvvVkZjemw=
Expand Down
4 changes: 3 additions & 1 deletion scripts/test-w-coverage.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@

set -aueo pipefail

readarray -t modules < <(go list ./... | grep -v tests/e2e)

go test -timeout 120s \
-failfast \
-v \
-coverprofile=coverage.txt \
-covermode count ./... | tee testoutput.txt || { echo "go test returned non-zero"; exit 1; }
-covermode count "${modules[@]}" | tee testoutput.txt || { echo "go test returned non-zero"; exit 1; }

# shellcheck disable=SC2002
cat testoutput.txt | go run github.com/jstemmer/go-junit-report > report.xml
Expand Down
108 changes: 108 additions & 0 deletions tests/e2e/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# E2E OSM Testing
eduser25 marked this conversation as resolved.
Show resolved Hide resolved
## Table of Contents
- [Overview](#overview)
- [Files and structure](#files-and-structure)
- [Running the tests](#running-the-tests)
- [Kind cluster](#kind-cluster)
- [Other K8s deployment](#other-k8s-deployment)
- [Flags](#flags)

## Overview
End-to-end tests verify the behavior of the entire system. For OSM, e2e tests will install a control plane, install test workloads and SMI policies, and check that the workload is behaving as expected.

## Files and structure
eduser25 marked this conversation as resolved.
Show resolved Hide resolved
OSM's e2e tests are located in `tests/e2e`.
The tests are written using Ginkgo and Gomega so they may also be directly invoked using `go test`. Be sure to build the `osm-controller` and `init` container images and `osm` CLI before directly invoking the tests ([see instructions below](#running-the-tests)).

OSM's framework has an init mechanism which will automatically initialize and parse flags and variables from both `env` and `go test flags` if any are passed to the test. The hooks for initialization and cleanup are set at Ginkgo's `BeforeEach` at the top level of test execution (between Ginkgo `Describes`); we henceforth recommend keeping every test in its own `Describe` section, as well as on a separate file for clarity. You can refer to [suite_test](tests/e2e/suite_test.go) for more details about the init process.

## Running the tests
Running the tests will require a running Kubernetes cluster. If you do not have a Kubernetes cluster to run the tests onto, you can choose to run them using `Kind`, which will make the test framework initialize a cluster on a local accessible docker client.

The tests can be run using the `test-e2e` Makefile target at repository root level (which defaults to use Kind), or alternatively `go test` targetting the test folder, which gives more flexibility but depends on related `env` flags given or parsed by the test.

Please refer to the [Kind cluster](#kind-cluster) or [Other K8s deployment](#other-k8s-eployment) and follow the instructions to setup potential env flags required by either option.

In addition to the flags provided by `go test` and Ginkgo, there are several custom command line flags that may be used for e2e tests to configure global parameters like container image locations and cleanup behavior. You can see the list of flags under the [flag section](#flags) below.

### Kind cluster
The following `make` target will create local containers for the OSM components, tagging them with `CTR_TAG`, and will launch the tests using Kind cluster. A Kind cluster is created at test start, and requires a docker interface to be available on the host running the test.
When using Kind, we load the images onto the Kind nodes directly (as opposed to providing a registry to pull the images from).
```
CTR_TAG=not-latest make test-e2e
```
Note: If you use `latest` tag, K8s will try to pull the image by default. If the images are not pushed to a registry accessible by the kind cluster, image pull errors will occur. Or, if an image with the same name is available, like `openservicemesh/init:latest`, then that publicly available image will be pulled and started instead, which may not be as up-to-date as the local image already loaded onto the cluster.

### Other K8s deployment
Have your Kubeconfig file point to your testing cluster of choice.
The following code uses `latest` tag by default. Non-Kind deployments do not push the images on the nodes, so make sure to set the registry accordingly.
```
export CTR_REGISTRY=<myacr>.dockerhub.io # if needed, set CTR_REGISTRY_USER and CTR_REGISTRY_PASSWORD
make build-osm
make docker-push
go test ./tests/e2e -test.v -ginkgo.v -ginkgo.progress
```

### Flags
#### (TODO) Kubeconf selection
Currently, test init will load a `Kubeconf` based on Defalut Kubeconf Loading rules.
If Kind is used, the kubeconf is temporarily replaced and Kind's kubeconf is used instead.

#### Container registry
A container registry where to load the images from (OSM, init container, etc.). Credentials are optional if the container registry allows pulling the images publicly:
```
-ctrRegistry string
Container registry
-ctrRegistrySecret string
Container registry secret
-ctrRegistryUser string
Container registry username
eduser25 marked this conversation as resolved.
Show resolved Hide resolved
```
If container registry user and password are provided, the test framework will take care to add those as Docker secret credentials for the given container registry whenever appropriate (tenant namespaces for `init` containers, OSM intallation, etc).
Container registry related flags can also be set through env:
```
export CTR_REGISTRY=<your_cr>.dockerhub.io
export CTR_REGISTRY_USER=<uername> # opt
export CTR_REGISTRY_PASSWORD=<password> # opt
```

#### OSM Tag
The following flag will refer to the version of the OSM platform containers (OSM and init) for test to use:
```
-osmImageTag string
OSM image tag (default "latest")
```
Make sure you have compiled the images and pushed them on your registry first if you are not using a kind cluster:
```
export CTR_REGISTRY=myacr.dockerhub.io
export CTR_TAG=mytag # Optional, 'latest' used by default
make docker-push-init docker-push-osm-controller. # Use docker-build-* targets instead when using kind
```

#### Use Kind for testing
Testing implements support for Kind. If `kindCluster` is enabled, a new Kind cluster will be provisioned and it will be automatically used for the test.
```
-kindCluster
Creates kind cluster
-kindClusterName string
Name of the Kind cluster to be created (default "osm-e2e")
-cleanupKindCluster
Cleanup kind cluster upon exit (default true)
-cleanupKindClusterBetweenTests
Cleanup kind cluster between tests (default true)
```

#### Test specific flags

Worth mentioning `cleanupTest` is especially useful for debugging or leaving the test in a certain state at test-exit.
When using Kind, you need to use `cleanupKindCluster` and `cleanupKindClusterBetweenTests` in conjunction, or else the cluster
will anyway be destroyed.
```
-cleanupTest
Cleanup test resources when done (default true)
-meshName string
OSM mesh name (default "osm-system")
eduser25 marked this conversation as resolved.
Show resolved Hide resolved
-waitForCleanup
Wait for effective deletion of resources (default true)
```
Plus, `go test` and `Ginkgo` specific flags, of course.
eduser25 marked this conversation as resolved.
Show resolved Hide resolved
Loading