From e6942864537cb8f70a99a2c407a63558131c3aa5 Mon Sep 17 00:00:00 2001 From: Jorge Turrado Ferrero Date: Fri, 18 Aug 2023 08:55:57 +0200 Subject: [PATCH] chore: Add testing documentation (#4884) * Add testing strategy document Signed-off-by: Jorge Turrado Signed-off-by: Jorge Turrado * Update TESTING.md Signed-off-by: Jorge Turrado * Update TESTING.md Signed-off-by: Jorge Turrado * add extra info to tests Signed-off-by: Jorge Turrado --------- Signed-off-by: Jorge Turrado Signed-off-by: Jorge Turrado --- README.md | 4 ++++ TESTING.md | 22 ++++++++++++++++++++++ tests/README.md | 7 +++++++ 3 files changed, 33 insertions(+) create mode 100644 TESTING.md diff --git a/README.md b/README.md index c7ee8bb5029..a2c864489cf 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ We are a Cloud Native Computing Foundation (CNCF) incubation project. - [Releases](#releases) - [Contributing](#contributing) - [Building & deploying locally](#building--deploying-locally) + - [Testing strategy](#testing-strategy) @@ -83,3 +84,6 @@ You can find contributing guide [here](./CONTRIBUTING.md). ### Building & deploying locally Learn how to build & deploy KEDA locally [here](./BUILD.md). + +### Testing strategy +Learn more about our testing strategy [here](./TESTING.md). diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 00000000000..67d465b37c6 --- /dev/null +++ b/TESTING.md @@ -0,0 +1,22 @@ +# Testing strategy + +## Unit tests / code coverage + +There are unit tests present for each scaler implementation, and for the majority of the core. +Code coverage “is something that we need to work on” constantly. + +However, using a code coverage tooling is not useful for the KEDA project in the current state given a lot of functionality is covered by end-to-end tests, which are not considered and thus our code coverage metrics will be misleading. + +For each PR, we automatically build and run our unit test suite but also build Docker images for both amd64 and arm64 architectures. As part of our CI process, we also perform various security checks for which you can learn more in our security section. + +Lastly, we automatically perform code quality analysis with golangci-lint and check licenses of our dependencies with FOSSA. + +## End-to-end tests + +There are end-to-end tests for the core functionality and majority of features of KEDA as well as the scalers that it offers. These tests are required for every PR and run in the CI (however, maintainers trigger them as a security precaution). Additionally, we run our e2e test suite for every merged commit to the main branch as well as during our nightly CI schedule ([link](https://github.com/kedacore/keda/actions/workflows/nightly-e2e.yml)). Implementing end-to-end tests is a requirement for adding a new scaler, as per [our policy](https://github.com/kedacore/governance/blob/main/SCALERS.md#requirements-for-a-built-in-scaler). + +The project runs two Kubernetes clusters on which all e2e tests are ran automatically. Microsoft Azure has donated a dedicated Azure subscription so that all maintainers can manage these cloud resources and allow us to run our automated tesing and automation. This is in addition to CNCF’s Cloud Credits program which we use to provision test resources in AWS & GCP for scaler e2e tests as well. + +Both the cluster management as well as the cloud resources required for our automated tests are managed by Terraform and [available on GitHub](https://github.com/kedacore/testing-infrastructure) so that every contributor can open a PR with the infrastructure changes that they require. Everything is automatically deployed by using GitHub Actions to ensure we are running the latest configuration and can easily migrate to other infrastructure, if we have to. + +Additionally, CNCF sponsors KEDA by providing arm64 machines on which we build our ARM64 image and Vexxhost sponsors an OpenStack instance to run tests on these as well. diff --git a/tests/README.md b/tests/README.md index bd478733369..76510581db5 100644 --- a/tests/README.md +++ b/tests/README.md @@ -16,6 +16,13 @@ go test -v -tags e2e ./scalers/... go test -v -tags e2e ./utils/cleanup_test.go # Skip if you want to keep testing. ``` +> **Note** +> As default, `go test -v -tags e2e ./utils/setup_test.go` deploys KEDA from upstream's main branch, +> if you are adding an e2e test to your own code, this is not useful as you need your own version. +> Like for [building and deploying your own image](../BUILD.md#custom-keda-as-an-image), you can use +> the Makefile envrionment variables to customize KEDA deployment. +> eg. `IMAGE_REGISTRY=docker.io IMAGE_REPO=johndoe go test -v -tags e2e ./utils/setup_test.go` + ### Specific test ```bash