Merge pull request #788 from fabriziosestito/test/parallel-execution

test: split envtest/real cluster tests

.github/workflows/ci.yml

@@ -36,7 +36,10 @@ jobs:
       - uses: actions/setup-go@cdcb36043654635271a94b9a6d1392de5bb323a7 # v5.0.1
         with:
           go-version: "1.22"
-      - run: make integration-tests
+      - name: Install gingko
+        run: go install github.com/onsi/ginkgo/v2/ginkgo
+      - run: make integration-tests-envtest
+      - run: make integration-tests-real-cluster
       - name: Upload integration-tests coverage to Codecov
         uses: codecov/codecov-action@e28ff129e5465c2c0dcc6f003fc735cb6ae0c673 # v4.5.0
         env:
@@ -59,9 +62,6 @@ jobs:
         uses: golangci/golangci-lint-action@a4f60bb28d35aeee14e6880718e0c85ff1882e64 # v6.0.1
         with:
           version: v1.57.2
-          # Disable caching as a workaround for https://github.com/golangci/golangci-lint-action/issues/135.
-          # The line can be removed once the golangci-lint issue is resolved.
-          skip-pkg-cache: true
 
   shellcheck:
     name: Shellcheck

CONTRIBUTING.md

@@ -42,22 +42,22 @@ You can use [k3d](https://k3d.io/) to create a local cluster for development pur
 ### Settings
 
 The `tilt-settings.yaml.example` acts as a template for the `tilt-settings.yaml`
-file that you need to create in the root of this repository.  Copy the example
-file and edit it to match your environment.  The `tilt-settings.yaml` file is
+file that you need to create in the root of this repository. Copy the example
+file and edit it to match your environment. The `tilt-settings.yaml` file is
 ignored by git, so you can edit it without concern about committing it by
 mistake.
 
 The following settings can be configured:
 
 - `registry`: the container registry to push the controller image to. If you
-don't have a private registry, you can use `ghcr.io` provided your cluster has
-access to it.
+  don't have a private registry, you can use `ghcr.io` provided your cluster has
+  access to it.
 
 - `image`: the name of the controller image. If you are using `ghcr.io` as your
-registry, you need to prefix the image name with your GitHub username.
+  registry, you need to prefix the image name with your GitHub username.
 
 - `helm_charts_path`: the path to the `helm-charts` repository that you cloned
-in the previous step.
+  in the previous step.
 
 Example:
 
@@ -84,6 +84,90 @@ empty cluster:
 tilt up --stream
 ```
 
+## Testing
+
+### Run tests
+
+Run all tests:
+
+```console
+make test
+```
+
+Run unit tests only:
+
+```console
+make unit-tests
+```
+
+Run controller integration tests only:
+
+```console
+make integration-tests
+```
+
+Run tests that do not require a Kubernetes cluster only:
+
+```console
+make integration-tests-envtest
+```
+
+Run tests that require a Kubernetes cluster only:
+
+```console
+make integration-tests-real-cluster
+```
+
+### Writing (controller) integration tests
+
+The controller integration tests are written using the [Ginkgo](https://onsi.github.io/ginkgo/)
+and [Gomega](https://onsi.github.io/gomega/) testing frameworks.
+The tests are located in the `internal/controller` package.
+
+By default, the tests are run by using [envtest](https://book.kubebuilder.io/reference/envtest) which
+sets up an instance of etcd and the Kubernetes API server, without kubelet, controller-manager or other components.
+
+However, some tests require a real Kubernetes cluster to run.
+These tests must be marked with the `real-cluster` ginkgo label.
+
+Example:
+
+```
+var _ = Describe("A test that requires a real cluster", Label("real cluster") func() {
+    Context("when running in a real cluster", func() {
+        It("should do something", func() {
+            // test code
+        })
+    })
+})
+```
+
+To run only the tests that require a real cluster, you can use the following command:
+
+```console
+make integration-test-real-cluster
+```
+
+The suite setup will start a [k3s testcontainer](https://testcontainers.com/modules/k3s/) and run the tests against it.
+It will also stop and remove the container when the tests finish.
+
+Note that the `real-cluster` tests are slower than the `envtest` tests, therefore, it is recommended to keep the number of `real-cluster` tests to a minimum.
+An example of a test that requires a real cluster is the `AdmissionPolicy` test suite, since at the time of writing, we wait for the `PolicyServer` Pod to be ready before reconciling the webhook configuration.
+
+### Focusing
+
+You can focus on a specific test or spec by using a [Focused Spec](https://onsi.github.io/ginkgo/#focused-specs).
+
+Example:
+
+```go
+var _ = Describe("Controller test", func() {
+    FIt("should do something", func() {
+        // This spec will be the only one executed
+    })
+})
+```
+
 ## Tagging a new release
 
 ### Make sure to update the CRD docs
@@ -133,24 +217,24 @@ everything works well:
 - [ ] Update audit scanner code
 - [ ] Run audit scanner tests or check if the CI is green in the main branch
 - [ ] Bump policy server version in the `Cargo.toml` and update the `Cargo.lock`
-file. This requires a PR in the repository to update the files in the main
-branch. Update the local code after merging the PR
+      file. This requires a PR in the repository to update the files in the main
+      branch. Update the local code after merging the PR
 - [ ] Run policy server tests or check if the CI is green in the main branch
 - [ ] Bump kwctl version in the `Cargo.toml` and update the `Cargo.lock` file.
-This requires a PR in the repository to update the files in the main branch.
-Update the local code after merging the PR
+      This requires a PR in the repository to update the files in the main branch.
+      Update the local code after merging the PR
 - [ ] Run kwctl tests or check if the CI is green in the main branch
 - [ ] Tag audit scanner
 - [ ] Tag policy server
 - [ ] Tag controller
 - [ ] Tag kwctl
 - [ ] Wait for all CI running in all the major components (audit scanner,
-controller, policy server and kwctl) to finish
+      controller, policy server and kwctl) to finish
 - [ ] Check if the Helm chart repository CI open a PR updating the Helm charts
-with the correct changes.
+      with the correct changes.
   - [ ] Check if the `kubewarden-controller` chart versions are correctly bumped
   - [ ] Check if the `kubewarden-defaults` chart versions are correctly bumped
   - [ ] Check if the `kubewarden-crds` chart versions are correctly bumped
   - [ ] Check if kubewarden-controller, kubewarden-defaults and kubewarden-crds
-  charts have the same `appVersion`
+        charts have the same `appVersion`
 - [ ] Check if CI in the Helm chart PR is green. If so, merge it.