[13] Integration Test Suite for Booster support

[miteshvp]

Description

As an OSIO Tester, I would like to increase UI Test coverage in support of all the boosters.

The boosters to be covered are:

• HTTP: Vertx, Springboot, WildflySwarm
• HealthCheck: Vertx, Springboot, WildflySwarm
• ConfigMap: Vertx, Springboot, WildflySwarm

Acceptance Criteria

Integration Tests (to exercise the Boosters running in the context of OSIO) should increase the overall coverage w.r.t boosters that a normal OSIO users perform.

Modularize the Test Structure as Single Test Suites

One difficulty in testing Boosters in OSIO is that each test must perform significant setup tasks such as the creation of a space and quickstart before a test can be run. The current e2e tests perform these tasks once for each test. This results in each test requiring several minutes to execute, and has each test return a single pass/fail result value.

For the Booster tests, we propose a different approach. While the Booster tests will exercise OSIO through its UI (as is the case with the e2e tests), the Booster tests will be organized in the following manner:

The tests will be divided into test suites. Each test will be include multiple tests.
The first test to be executed will perform all the set up steps required by the test suite.
The remaining tests will be structured as atomic units that can be run in any order.
At the conclusion of the test suite, any resources created by the tests will be deleted.

Actions to be performed by the tests

• Login
  • all test cases
• Create a workspace
  • Integration Test for Booster support: Setup #2568 Setup
• Add to space by selecting Import a Github Repo option
  • Integration Test for Booster support: Import a Github repo #2829 Import a Github repo
• Create a new application using the new launcher wizard
  • Integration Test for Booster support: Setup #2568 Setup
• Select Github Repo to import into
  • Integration Test for Booster support: Setup #2568 Setup
• Select the Pipelines
  • Integration Test for Booster support: Pipeline #2571 Pipeline
  • Integration Test for Booster support: Import a Github repo #2829 Import a Github repo
  • Integration Test for Booster support: Check for Stack/Analytic reports #2573 Check for Stack/Analytic reports
• Triggers the CD Build (Jenkins)
  • Integration Test for Booster support: Triggers the CD Build (Jenkins) #2572 Triggers the CD Build (Jenkins)
• Check for Build success
  • Integration Test for Booster support: Pipeline #2571 Pipeline
  • Integration Test for Booster support: Import a Github repo #2829 Import a Github repo
• Verify deployment to stage and run, verify that ends can be reached
  • Integration Test for Booster support: Pipeline #2571 Pipeline
  • Integration Test for Booster support: Import a Github repo #2829 Import a Github repo
• Make change to app, create PR, verify changes are deployed to run and stage Pipeline fails to deploy changes to an application #2505
  • Integration Test for Booster support: Triggers the CD Build (Jenkins) #2572 Triggers the CD Build (Jenkins)
• Check for Stack reports
  • Integration Test for Booster support: Check for Stack/Analytic reports #2573 Check for Stack/Analytic reports
• Open Openshift Console from OSIO
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
• Copy Login Command (under username right hand side)
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
• Open Che workspace
  • Integration Test for Booster support: Setup #2568 Setup
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
  • Integration Test for Booster support: Build the project using Maven in the Che terminal window #2574 Build the project using Maven in the Che terminal window
  • Integration Test for Booster support: Verify that the preview URL provided in the Che terminal is correct and can be accessed #2575 Verify that the preview URL provided in the Che terminal is correct and can be accessed
  • Integration Test for Booster support: Run the project from the Che menu #2577 Run the project from the Che menu
  • Integration Test for Booster support: Debug the project from the Che menu #2578 Debug the project from the Che menu
  • Integration Test for Booster support: Run the project's Junit tests from the Che menu #2579 Run the project's Junit tests from the Che menu
  • Integration Test for Booster support: Modify the project source code in Che - verify rebuild, redeploy #2580 Modify the project source code in Che - verify rebuild, redeploy
• Check the success of importing github contents by opening a manifest file (pom.xml)
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
• Paste login command copied in step(j) inside of che terminal to login to Openshift
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
• Switch to user namespace by doing oc project command
  • Integration Test for Booster support: Access project in OSO #2570 Access project in OSO
• Build the project using mvn clean install -Popenshift in the Che terminal window
  • Integration Test for Booster support: Build the project using Maven in the Che terminal window #2574 Build the project using Maven in the Che terminal window
• Verify that the preview URL provided in the Che terminal is correct and can be accessed
  • Integration Test for Booster support: Verify that the preview URL provided in the Che terminal is correct and can be accessed #2575 Verify that the preview URL provided in the Che terminal is correct and can be accessed
  • Integration Test for Booster support: Modify the project source code in Che - verify rebuild, redeploy #2580 Modify the project source code in Che - verify rebuild, redeploy
  • Integration Test for Booster support: Run the project from the Che menu #2577 Run the project from the Che menu
• Run the project from the Che menu
  • Integration Test for Booster support: Run the project from the Che menu #2577 Run the project from the Che menu
• Debug the project from the Che menu
  • Integration Test for Booster support: Debug the project from the Che menu #2578 Debug the project from the Che menu
• Run JUnit tests from the Che menu
  • Integration Test for Booster support: Run the project's Junit tests from the Che menu #2579 Run the project's Junit tests from the Che menu
• Modify the project source code in Che - verify rebuild, redeploy
  • Integration Test for Booster support: Modify the project source code in Che - verify rebuild, redeploy #2580 Modify the project source code in Che - verify rebuild, redeploy
• Commit and push the project source code to git repo - verify rebuild, redeploy
  • Integration Test for Booster support: Modify the project source code in Che - verify rebuild, redeploy #2580 Modify the project source code in Che - verify rebuild, redeploy
• Deletion or Reset Environment
  • Integration Test for Booster support: Setup #2568 Setup
  • Integration Test for Booster support: Clean up #2581 Clean up
• Logout
  • All test cases

Test Code Repo

Booster tests will be managed in the ee_test directory of this repo: https://github.com/fabric8io/fabric8-test/tree/master/ee_tests - and will leverage/extend the EE test framework.

Pull Requests

• POC test for Junit tests in Che - fix(test): poc test for junit tests running in Che fabric8io/fabric8-test#509
• POC test for running Maven build in Che - fix(test): Initial test for executing test from mvn in Che workspace terminal shell fabric8io/fabric8-test#463
• POC tets for Che editor in OSIO - Editor che test POC  fabric8io/fabric8-test#513
• Test for the new launcher UI:
  • ee_tests: PoC for e2e testing of the new launcher UI fabric8io/fabric8-test#522
  • ee_tests: ngx_launcher - Implement the whole quickstart flow. fabric8io/fabric8-test#562
• Integration Test Suite for Boosters - ee_test: booster integration tests skeleton fabric8io/fabric8-test#527
• Configure new Booster E2E tests to run daily on Centos CI - Configure new Booster E2E tests to run daily on Centos CI fabric8io/fabric8-test#552

[ldimaggi]

Test to create a new Che workspace is running now:
https://ci.centos.org/view/Devtools/job/devtools-test-e2e-openshift.io-chetest-cluster-one/

[ldimaggi]

Test to run mvn build command in Che / OSIO - fabric8io/fabric8-test#463

[miteshvp]

@ldimaggi - I have edited this issue to consider modularized test suites as discussed in our last meeting. Please let me know if there are any missing links or pieces to be added

[ldimaggi]

Test to run junit tests in Che for booster - fabric8io/fabric8-test#509

[ldimaggi]

My thinking for the structure of the tests follows - and, by the way, I think that we should start to consider and refer to these tests not as E2E tests but as Booster-OSIO tests as the primary goal of the tests is to verify the operation of the Boosters in the context of OSIO.

Each Booster will be verified by the same basis set of tests. We may additionally create Booster-specific tests as needed.

The tests should be organized into test suites. The first case test in each suite will perform setup tasks and the final case test will perform tear down tasks. The remaining test cases will be structured such that they can be run in any combination and that any subset of the tests can be run.

Each test case will start with the test acocunt logging into OSIO and end with the test account logging out. Each test suite run will utilize a single OSIO userspace and Che workspace, both of which will be created during the test setup. We can use the same space and Che workspace name for each test suite execution as the workspace and space will be deleted as part of the test teardown.

For the short term (sprint 146) the starting point for the execution of the tests will be after the Boosters have been installed into the test user account. Longer term, the tests will install the Boosters by using the new launcher.

Test case #2568: Setup
Steps: Login, reset the user environment, create a new boosters, space, Che workspace, logout.

Test case #2570: Access project in OSO
Steps: Login, then:
Open Openshift Console from OSIO
Copy Login Command (under username right hand side)
Open Che workspace
Check the success of importing github contents by opening a manifest file (pom.xml)
Paste login command copied in step(j) inside of che terminal to login to Openshift
Switch to user namespace by doing oc project command
Logout

Test case #2571: Pipeline
Steps: Login, verify the completion of the build pipeline and deployment to stage and run. Verify successful operation of deployed booster at stage and run endpoints., logout.

Test case #2572: Triggers the CD Build (Jenkins),
Commit and push the project source code to git repo - verify rebuild, redeploy
Steps: Login, modify source code, create PR, verify redeployment to stage and run, verify changes are deployed, logout.

Test case #2573: Check for Stack/Analytic reports
Steps: Login, verify that build completion creates stack report, verify report contents by compariing to expected results, logout.

Test case #2574: Build the project using mvn clean install -Popenshift in the Che terminal window
Steps: Login, open Che workspace, execute mvn clean install command in Che terminal window, logout.

Test case #2575: Verify that the preview URL provided in the Che terminal is correct and can be accessed
Login, open Che workspace, execute mvn clean install command in Che terminal window, verify URL generated endpoint is reachable and that the deployed Booster functions correctly, logout.

Test case #2577: Run the project from the Che menu
Steps: Login, open Che workspace, run the Booster from the Che run menu, verify results, logout.

Test case #2578: Debug the project from the Che menu
Steps: Login, open Che workspace, debug the Booster from the Che run menu, verify results, logout.

Test case #2579: Run the project's Junit tests from the Che menu
Steps: Login, open Che workspace, run the Booster Junit tests from the Che run menu, verify results, logout.

Test case #2580: Modify the project source code in Che - verify rebuild, redeploy
Steps: Login, open Che workspace, edit source code, verify rebuild and redeploy, verify redeployment to stage and run, verify changes are deployed, logout.

Test case #2829: Import a Github repo
Steps: Login, add to space by importing existing code, verify the completion of the build pipeline and deployment to stage and run. Verify successful operation of deployed booster at stage and run endpoints., logout.

Test case #2581: Teardown
Steps: Login, reset user environment, verify environment is cleaned, logout.

[ppitonak]

1. As you wrote in your description, Setup and Teardown are not test cases but setup/tear down tasks.
2. It's unnecessary to list login and logout in every test case, see ^
3. What I'm missing most is the fact that these type of tests should not depend on production environment. We should think about how to test booster+one OSIO component outside of prod/prod-preview environment.

[pmacik]

@ppitonak That (3) would be very tricky since OSIO is an integration of many components KC, RHD, Auth etc. Our goal is to test the integration of boosters into OSIO. In order to test it all outside prod/prod-preview one would need to re-create the whole setup (and/or mock some components) which would be impractical.
@ldimaggi WDYT?

[sunilk747]

IMO it would be better to create userspace for a booster once and use it for further testing. Creating userspace for each test would be time taking and kind of redundant.
@ldimaggi @pmacik What are your opinions on this.

[pmacik]

@sunilk747 Creating a userspace is not done for each test. It is done just once for the whole suite in the "Test case: Setup" and re-used for the rest of the tests.

[hrishin]

refer #2809

[ldimaggi]

See: #3566

[pmacik]

Moved tracking to https://openshift.io/openshiftio/openshiftio/plan/detail/2405.