Set up PR Check for Java Che 7 plugins

[tsmaeder]

We now have the ability to write tests for a particular VS Code extension (#14409). The goal of this task is to:

• Write a test suite that smoke-tests the VS Code Java extension
• Set up a CI task that runs that test suite as a PR Check whenever someone changes a plugin that uses VS Code Java.

[JPinkney]

Just a few things left:

• Implementation of the languages test api needs to be merged: PR
• Centos CI languages test PR check needs to be enabled for che-plugin-registry: PR
• The Java tests should probably be moved to another location: Repo
• The che-plugin-registry PR check will need to be tested on Centos CI: PR

[JPinkney]

After having the discussion on both the Theia and Che community calls I think we might end up needing a slight change of direction. I think what we need to do is utilize the built-in commands in order to make the calls to the language server, rather than maintaining the extra API.

This has a few advantages:

• We don't have to maintain extra api
• We can basically re-use vscode api syntax
• We don't have to decide which namespace it should live in 😉

But has the disadvantage that we have to wait for all that api to be implemented before we can proceed (I think only a couple are implemented on the theia side).

The only issue I can foresee is that I'm not entirely sure what would happen if you call a command such as: vscode.executeWorkspaceSymbolProvider

before workspace symbol is even added by a language server. My gut feeling is that the tests would just fail. In that case, I think we can adapt the current open PR in che-theia to wait for the language server feature to be added before making the request (similar to what the PR does right now).

@tsmaeder @benoitf If this is the way to proceed I can update the che-theia PR to wait for the language server feature to be added before making the request and modify the sample tests to use the few vscode built-in commands that are available in theia so far.

[JPinkney]

Here's a little overview of how everything works if someone wants to pick it up:

How the java tests work now

-> once the java tests are cloned
-> the tests wait for vscode-java activation
-> then the tests will use vscode built-in commands to call different language methods
-> check the responses
-> write the test result to /projects/test.log
-> once the tests are finished /projects/test.log will contain either TESTS_PASSED or TESTS_FAILED

How the scripts should work

In terms of how the tests should be done, I've created a flow diagram [1]. The current problem is that the tests get stuck on workspace is opened in browser [2]. As it stands, the way that is done is by using python with selenium to open the workspace url, login to keycloak, and then follow the redirect to theia (it does this all headless). The reason this is needed is that a workspace needs to be opened in the browser before git clone will run and the tests executed. The script works locally but running it on the centos ci leads to the python script finishing with:

1588710026238	Marionette	DEBUG	Closed connection 0
JavaScript error: https://static.developers.redhat.com/che/theia_artifacts/theia.9baa83d852b65a9b3143.js, line 1: Error: connection is closed

It's not entirely clear why that issue is occurring and why running that python script works locally but not on centos ci. If you want to try out running the tests automatically you can start a workspace with: https://raw.githubusercontent.com/JPinkney/Che-Java-Tests/master/devfile.yaml

[1]

[2] - https://github.com/eclipse/che-plugin-registry/blob/2b877e739dbfdfad4d4642b0d2f3d8f71086cd36/.ci/functional-tests-utils.sh#L318

[ericwill]

Thanks Josh, the Plugins Team be taking over this work starting in the next sprint (184).

[ericwill]

This is also relevant to #17027

[benoitf]

Is it planned to use github action to run the tests .

[ericwill]

Is it planned to use github action to run the tests .

I believe the first iteration is using jenkins, but we can use GH actions in the next iteration.