integration-tests

This repository contains files related to integration tests for devfile.

NOTE: All related issues are being tracked under the main devfile API repo https://github.com/devfile/api with the label area/integration-tests

Daily CI integration tests

• Github actions workflows is configured to run nightly integration tests and also when a PR is opened against 'main' branch.
  • Devfile library/API Go tests
• Additional integration tests are configured with OpenShift CI system to run ODO and ODC tests with devfile.

ODO tests on OpenShift CI

ODO integration test cases

OCP version	Prow Test Status
4.15	devfile-integration-tests-main-v4.15.odo
4.12	devfile-integration-tests-main-v4.12.odo
4.11	devfile-integration-tests-main-v4.11.odo

ODC tests on OpenShift CI

ODC integration test cases

OCP version	Prow Test Status
4.15	devfile-integration-tests-main-v4.15.console
4.12	devfile-integration-tests-main-v4.12.console
4.11	devfile-integration-tests-main-v4.11.console

How to run integration tests on a local machine

NOTE: This section covers the required test environment for macOS specifically, however the similar steps can be used for other OSes.

How to run library/API integration tests on a local machine

NOTE: This test is based on devfile library tests

1. clone the repository
2. cd integration-tests
3. Run Make test
4. The test creates the following files:
   1. ./tmp/test.log contains log output from the tests.
   2. ./tmp/library_test/Test_*.yaml and ./tmp/integration_test/Test_*.yaml are the devfiles that are randomly generated at runtime. The file name matches the name of the test function which resulted in them being created.
   3. If a test detects an error when comparing properties returned by the parser with expected properties
      • ./tmp/library_test/Test_*_<property id>_Parser.yaml - property as returned by the parser
      • ./tmp/library_test/Test_*_<property id>_Test.yaml - property as expected by the test

NOTE: each run of the test removes the existing contents of the ./tmp directory

[Deprecated] How to run Pytest-based ODO tests on a local machine

Prerequisites

• Python 3.9.10
• pipenv : run pip install --user pipenv
• Minikube or OpenShift
• odo

Run tests

1. clone the repository
2. cd integration-tests
3. run pipenv install --dev
4. Start Minikube or OpenShift (e.g. crc)
5. run pipenv run pytest tests/odo -v to test all, or pipenv run pytest tests/odo/<target test>.py -v to test target test cases.
6. [Optional] In order run integration tests with the latest Odo build
   1. Build and install by following the instruction on 'odo.dev'
   2. run pipenv run pytest tests/odo_300 -v to test all, or pipenv run pytest tests/odo_300/<target test>.py -v to test target test cases.

How to run OpenShift CI ODO tests on a local machine

ODO integration tests

Prerequisites

• Go 1.21 and Ginkgo latest version
• git
• OpenShift Cluster. e.g. crc environment for 4.* local cluster
• [Optional] xunit-viewer and jrm : required to get performance test results in a merged html format in addition to junit*.xml.

Run tests

1. cd local/odo
2. run ./odo-integration-tests.sh : it runs odo catalog command test by default. In order to run other test cases, modify ./odo-integration-tests.sh by enabling other test options. e.g. make test-cmd-devfile-create

How to run performance tests for ODO

1. Open Makefile and remove --skipMeasurements option from GINKGO_FLAGS_ALL flag.

2. Add Ginkgo's Measure block for the spec that you want to measure the performance. A sample Measure block can be found from https://github.com/devfile/integration-tests/tree/main/local/odo/tests/integration/devfile/cmd_devfile_catalog_test.go

3. [Optional] In order to get test results in html format, uncomment jrm... and xunit-viewer... command calls from odo-integration-tests.sh.

4. run ./odo-integration-tests.sh

   A sample performance test output in console.

   

   A sample test output in html format

   

How to run OpenShift CI ODC tests on a local machine

Tests in this repository are based on ODC integration tests.

Prerequisites

1. node.js >= 14 & yarn >= 1.20
2. go >= 1.21+
3. oc or kubectl and an OpenShift or Kubernetes cluster. In this document, CRC is used.
4. jq (for contrib/environment.sh)
5. Google Chrome/Chromium or Firefox for integration tests
6. Cypress - integration tests are implemented in Cypress.io.

Run tests

1. install CRC

2. run crc setup

3. run crc start and record credentials for kubeadmin. You may obtain the credentials by running crc console --credentials

4. run crc oc-env to configure your shell

5. git clone OpenShift console repository

6. build console. Login as kubeadmin and start a local console.

   cd console  
   ./build.sh        # Backend binaries are output to `./bin`
   oc login -u kubeadmin -p <kubeadmin_password>  
   source ./contrib/oc-environment.sh
   ./bin/bridge
   

   The console will be running at http://localhost:9000

7. Update console/frontend/packages/dev-console/integration-tests/features/addFlow/create-from-devfile.feature by adding TAG (@smoke) for scenarios to run tests.

8. Launch Cypress test runner

   cd console/frontend
   yarn run test-cypress-dev-console
   

9. click create-from-devfile.feature to run test

For more detail, see https://github.com/openshift/console#integration-tests.