E2E tests suite for Che 7

[rhopp]

As per latest discussions we decided, that best approach for E2E UI testing of Eclipse Che 7 would be to implement "happy path" suite:

• This suite will try to cover as much ground possible, but will not test any component too deeply.
• It should be based on real E2E user scenarios (new user, user with existing codebase&apps, ...)
• This suite should run on as many possible (supported) environments and product configs (pvc strategy, http/https, ...)
• This suite shouldn't be running for too long, this will allow it to be used as quality gate to eclipse/che repository.

E2E Test Script

1 - Deploy Che on minikube using the operator with authentication enabled

Steps

• Install minikube
• Pre-pull docker images needed by the workspace:
  • eclipse/che-operator:latest
  • registry.access.redhat.com/rhscl/postgresql-96-rhel7
  • registry.access.redhat.com/redhat-sso-7/sso72-openshift
  • eclipse/che-server:nightly
  • busybox
  • centos:centos7
  • eclipse/che-init-plugin-broker:v0.17
  • eclipse/che-unified-plugin-broker:v0.17
  • mariolet/petclinic:d2831f9b
  • centos/mysql-57-centos7
  • eclipse/che-machine-exec
  • eclipse/che-theia:next
  • eclipse/che-remote-plugin-runner-java8:next
  • maven:3.6.0-jdk-11
• Deploy Che: run chectl server:start --installer=operator
• Create a new Che user and login as that user

Success criteria

• Minikube status is running
• Che, Keycloak and Postre are Running
• It's possible to login as a new user and access the dashboard
• Images are pulled

Blockers and workarounds

• The operator installer for chectl is not working currently Chectl server:start command uses old plugin registry env var in configmap che-incubator/chectl#92. Helm (chectl server:start) can be used instead.
• Che server is not completely ready when chectl returns Improve che.isCheServerReady() method (flapping route/ingress issue) che-incubator/chectl#145
• Che is not accessible after being started by chectl on Kubernetes Che is not accessible after being started by chectl on Kubernetes #13840

2 - Generate a devfile from a running application

Steps

• 1 - Deploy Petclinic:

kubectl create namespace prod
kubectl apply -n prod -f https://gist.githubusercontent.com/l0rd/52eda1d1b57878a218cba17c0c93a477/raw/d3f962bdf373406eb57be20efcdea2e4da8b19f9/petclinic.yaml

• 2 - Generate a devfile for the development of that application:

chectl devfile:generate \
        --namespace=prod \
        --selector="app.kubernetes.io/name=petclinic" \
        --language=java \
        --git-repo='https://github.com/spring-projects/spring-petclinic.git' \
        --plugin='{"type": "chePlugin", "id": "redhat/vscode-yaml/0.4.0"}' > petclinic-devfile.yaml

Success criteria

• The CRUD application is responding
• A valid devfile is generated

Blockers and workarounds

• Improve devfile:generate command che-incubator/chectl#100

3 - Start a Workspace from a devfile

Steps

• 1 - Start a workspace using the devfile generated at step 2.2 using command

chectl workspace:start --devfile=petclinic-devfile.yaml

Success criteria
- Workspace started successfully
- Workspace is started in less than X seconds
- Project is cloned
- JDT LS and Java debug adapter are initialized correctly
- YAML LS is initialized correctly
- Kubernetes tooling is initialized correctly
- GitHub PR plugin is initialized correctly
- Terminal work

Blockers and workarounds

• It's not possible use the devfile generate above because we cannot generate a devfile with multiple kubernetes components yet ("Support specifying multiple selectors" in issue Improve devfile:generate command che-incubator/chectl#100). As a workaround use this devfile:

chectl workspace:start --devfile=https://gist.githubusercontent.com/Ohrimenko1988/93f5426f4ebc1705c55feb8ff0396a49/raw/19f626e545372d8607514d88aa13ef0ab03b144c/yaml-ls-bug.yaml

• GitHub PR plugin is not available yet Include existing GitHub plugin in Che Theia IDE #11867
• Yaml plugin fails to start on Kubernetes infrastructure Yaml plugin fails to start on Kubernetes infrastructure #13465
• Java LS "Classpath is incomplete" warning when loading petclinic Java LS "Classpath is incomplete" warning when loading petclinic #13427
• Unexpected errors during workspace creation based on Pet clinic project Unexpected errors during workspace creation based on Pet clinic project  #13498
• Happy Path test is sometimes failing on "Error highlighting" step Happy Path test is sometimes failing on "Error highlighting" step #13865

4 - Use Java IDE features and the inner loop

Steps

• 1 - Open a Java file, change some code and verify that error highlighting, code completion and code navigation work
• 2 - Use the "Build" task and verify that the application is build successfully
• 3 - Use the "Run" task and and verify that the application is updated successfully
• 4 - Set a breakpoint and use the "Debug" task to verify that the application stops at the breakpoint

Success criteria

• Java Code completion, error highlighting and code navigation should work successfully
• The provided "Build" task executes successfully
• The provided "Run" task executes successfully and a pop-up ask to open the application URL
• Changes done at step 4.1 should be applied
• Java Debugger should be able to attach successfully when executing the "Debug" task.
• Java Debugger should be stop at a breakpoint and resume application execution when clicking on resume.

Blockers and workarounds

• Warnings when loading petclinic Java LS "Classpath is incomplete" warning when loading petclinic #13427: as a workaround right click on the java folder from the project explorer and select "Add folder to Java source path"
• No run tasks and debug launch configuration are configured yet
• Source code is not mounted in the petclinic container, probably related to Support mountSources for non-plugin tools in devfile #12554
• Debug session can't be launched in the Theia editor Debug session can't be launched in the Theia editor #13576
• Running terminal process can't be opened or terminated by IDE top menu Running terminal process can't be opened or terminated by IDE top menu #13590
• At the launch of the workspace, the IDE "GWT" is loaded instead of "Theia" IDE At the launch of the workspace, the IDE "GWT" is loaded instead of "Theia" IDE #13681
• (Theia repo issue) Can not run Configured task Can not run Configured task eclipse-theia/theia#5064
• Building the project raises an OOM error. As a workaround patch the cm before starting the workspace:

kubectl --namespace=che patch cm che --patch '{"data":{"CHE_WORKSPACE_JAVA__OPTIONS": "-Xmx1500m", "CHE_WORKSPACE_MAVEN__OPTIONS": "-Xmx1500m"}}'

5 - Outer loop

Steps

• 1 - Create a local git branch, commit locally, push on a remote branch and open a GitHub PR using the GitHub plugin
• 2 - Merge the Pull request and verify that the build of the Docker image is started
• 3 - Inject the Kubernetes Context in the workspace
• 4 - Once the build completes use Che Kubernetes Tooling to perform a rolling update of the application

Success criteria

• A Pull Request can be created using the GitHub plugin
• Once the PR is merged a new image should be built
• The kubernetes context is inected successfully and it's possible to read/create k8s objects in production namespace
• The rolling update done with the Kubernetes Tooling Plugin is run successfully
• The application in production is updated successfully

Blockers

• No blockers

6 - Workspace sharing

Steps

• 1 - Create a factory from the current workspace
• 2 - Logout and login as a second user (without stopping the first workspace)
• 3 - Start the workspace as a second user using the factory link

Success criteria

• It is possible to create a factory
• Logout and login as another user are successful
• The workspace is started successfully using the factory (see criterias of STEP 3)

Blockers

• For now (CHE 7.0.0-RC-2.0-SNAPSHOT) it is not possible to create a factory from workspace that was created from devfile. Only workspaces created from configuration are supported..

7 - Persistence and workspace status restoring

Steps

• 1 - Modify some code stop/start the workspace and verify that the modifications are still there
• 2 - Stop one of the workspaces
• 3 - Stop/Start Che with chectl and verify that the workspaces status are recovered correctly (1 running and 1 stopped)

Success criteria

Blockers

• No blockers

8 - Uninstall

Steps

• 1 - Stop and delete both workspaces
• 2 - Stop and Uninstall Che using chectl and verify that all the resources have been free up

Success criteria

Blockers

• No blockers

Subtasks:

• - Implement first E2E suite for Che 7 Implement first E2E suite for Che 7 #12727
• - Configure CI for new E2E test suite Configure CI for new E2E test suite #12729

[l0rd]

As mentioned by @gorkem we should make sure that the happy path scenario take into account Che workshop scenario as well

[dmytro-ndp]

FYI: links to various samples for Java projects could be found in description of issue #12620

[l0rd]

@Ohrimenko1988 @musienko-maxim @rhopp I have updated the description of the e2e steps in this issue and the blockers/workaround I am aware of. What I ask you is to:

• Review the first 4 steps and verify that they correspond with the tests you are implementing. If not please comment so that we can discuss and update.
• Check [x] the steps that you have already implemented
• Add blockers and workarounds that are not already listed

[rhopp]

@l0rd As we would like to adapt the (slightly modified) happy path scenario also for che saas offering (che.openshift.io), I've tried to create workspace from the devfile mentioned in this issue.. This failed, with not enough resources (I was kinda expecting that).

(combined from similar events): Error creating: pods "workspacelfm70rbloln3cf4v.workspace-c6bf9f5c6-kg29l" is forbidden: [maximum cpu usage per Pod is 6, but limit is 8711m., maximum memory usage per Pod is 3Gi, but limit is 4679041024.]

Should we use something different for happy path test on saas che, or should we raise this with rh-che team and think about solution there?

[l0rd]

Good point @rhopp. Let me try to create a "lighter" devfile for that purpose.

[slemeur]

@rhopp : Is the scope for Che 7 GA been covered? so we can check that from the end game plan and move forward?

[rhopp]

Happy path was implemented and is being run as a PR check on multiple repositories.
I'm closing this one... For any future improvements, let's have separate issues.