CertSuite QE

Objective

The repository contains a set of test cases that run different test scenarios from certsuite project and verifies if these scenarios behave correctly under different environment conditions.

The certsuite-qe project is based on golang+ginkgo framework.

certsuite-qe project triggers the same test scenario from certsuite several times using different pre-configured OCP environment.

Once the triggered scenario is completed, the test case processes the report and verifies that the scenario is completed with the excepted result: skip/fail/pass.

Requirements

The tests are run on the OCP cluster with certain requirements that are listed below.

	Conditions	Mandatory
OCP Cluster	Version: >= 4.12, Node Count >= 3 with 2 cnf-worker nodes	Yes
Installed Operators	Performance Addon, Machine-config-operator	Yes
	Machine config pool, PTP operator, SR-IOV operator	No

Bare-minimum requirements consists of a OCP cluster with 3 nodes where 2 are cnf-worker nodes and 1 worker node.

Overview

The following test features are can run selectively or altogether.

• accesscontrol
• affiliatedcertification
• lifecycle
• manageability
• networking
• observability
• platformalteration
• performance
• operator
• preflight

Choose the variant that suits you best:

make test-features - will only run tests for the features that were defined in the FEATURES environment variable make test-all - will run the test suite for all features

Environment variables

The following environment variables are used to configure the test setup.

Env Variable Name	Purpose
FEATURES	To select the test scenarios that you are going to test, comma separated
CERTSUITE_REPO_PATH	Points to the absolute path to certsuite on your machine
CERTSUITE_IMAGE	Links to the Certsuite image. Default is quay.io/redhat-best-practices-for-k8s/certsuite
CERTSUITE_IMAGE_TAG	image tag that is going to be tested. Default is latest
TEST_IMAGE	Test image that is going to be used for all test resources such as deployments, daemonsets and so on. Default is quay.io/testnetworkfunction/k8s-best-practices-debug
DEBUG_CERTSUITE	Generate Debug folder that will contain Certsuite suites folders with Certsuite logs for each test.
CERTSUITE_LOG_LEVEL	Log level. Default is 4
DISABLE_INTRUSIVE_TESTS	Turns off the intrusive tests for faster execution. Default is false.
ENABLE_PARALLEL	Enable ginkgo -p parallel flags (experimental). Default is false.
FORCE_DOWNLOAD_UNSTABLE	Force download the unstable image. Default is false.
NON_LINUX_ENV	Allow the test suites to run in a non Linux environment. Default is false.

Steps to run the tests

Pre-requisites

Make sure docker or podman is running on the local machine.

Set your local container runtime to your environment with:

export CERTSUITE_CONTAINER_CLIENT=docker

Clone the repo and change directory to the cloned repo

git clone https://github.com/redhat-best-practices-for-k8s/certsuite-qe.git
cd certsuite-qe

Download and install needed dependencies

make install

Execute tests

• To run all tests

# Mac user
  DOCKER_CONFIG_DIR=$HOME/.docker \
  KUBECONFIG=$HOME/.kube/config \
  NON_LINUX_ENV= \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-all

# Linux user
  KUBECONFIG=$HOME/.kube/config \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-all

# Linux user with force download unstable image
 \
  FORCE_DOWNLOAD_UNSTABLE=true \
  KUBECONFIG=$HOME/.kube/config \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-all

• To run a specific test-suite:

# Mac user
  DOCKER_CONFIG_DIR=$HOME/.docker \
  FEATURES=platformalteration \
  KUBECONFIG=$HOME/.kube/config \
  NON_LINUX_ENV= \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-features

# Linux user
  FEATURES=platformalteration \
  KUBECONFIG=$HOME/.kube/config \
  DOCKER_CONFIG_DIR=$HOME/.docker \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-features

• To debug

Use DEBUG_CERTSUITE=true and CERTSUITE_LOG_LEVEL=debug while running the above commands. This would create a Debug folder containing suites folders with Certsuite logs for each of the tests.

# Mac user
  DEBUG_CERTSUITE=true \
  DOCKER_CONFIG_DIR=$HOME/.docker \
  FEATURES=platformalteration \
  KUBECONFIG=$HOME/.kube/config \
  NON_LINUX_ENV= \
  CERTSUITE_LOG_LEVEL=debug \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-features

# Linux user
  DEBUG_CERTSUITE=true \
  FEATURES=platformalteration \
  KUBECONFIG=$HOME/.kube/config \
  CERTSUITE_LOG_LEVEL=debug \
  CERTSUITE_REPO_PATH=$HOME/path/to/certsuite \
  make test-features

Running the unit tests

To execute the unit tests in the repository, run the following:

make test

Test exceptions on local kind cluster

• access-control-security-context
• affiliated-certification-container-is-certified-digest
• affiliated-certification-operator-is-certified
• platform-alteration-tainted-node-kernel

Nightly Runs Against Various Environments

The QE repo is being used in nightly automated runs in the following files:

• QE via Kind (Github Hosted)
• QE via OCP (Intrusive)
• QE via OCP (Non-Intrusive)

Contribution Guidelines

Fork the repo, create a new branch and create a PR with your changes.

License

CertSuite QE is copyright Red Hat, Inc. and available under an Apache 2 license.