Need to make sure that all devfiles from registry work on che.prod-preview.openshift.io

[ibuziuk]

Cc: https://github.com/redhat-developer/che-functional-tests/issues/543 (migrating this issue to rh-che)

Dev-file registry - https://che-devfile-registry.openshift.io/devfiles/

Stack images Acceptance Criteria

These are the things that we should verify to validate a particular stack

• workspace containers can be started
• terminal shell for the runtime container(s) is bash (git PS1 is nice to have)
• HOME is set
• No "User has no name" problem
• devfile commands can be executed successfully

Stack Language Servers Acceptance Criteria

• code completion works successfully
• code navigation works successfully
• debugging works successfully

Known issues

https://gist.github.com/amisevsk/86ec3540da1b9b5ccd98aa23d4e9fafb

[ibuziuk]

Also, I noticed that devfiles from prod-preview registry are not compatible with CR1, which is currently deployed on production:

Error occurred during creation a workspace from devfile located at https://che-devfile-registry.prod-preview.openshift.io/devfiles/dotnet/devfile.yaml. Cause: Devfile schema validation failed. Error: (/apiVersion):The object must not have a property whose name is "apiVersion".(/metadata):The object must not have a property whose name is "metadata".The object must have a property whose name is "specVersion".The object must have a property whose name is "name".

[amisevsk]

I've done a runthrough [1] of current devfiles on our dev-cluster (so not a full check on che.openshift.io) but a number of issues have been resolved in a general sense. The arbitrary-users patch PR has been merged and nightly images are available (in all cases except for the angular stack, which is based off alpine). This should resolve most known issues with the stacks, but should be verified.

[1] https://gist.github.com/amisevsk/86ec3540da1b9b5ccd98aa23d4e9fafb

[ibuziuk]

thanks, @amisevsk for the detailed Testing devfiles document with issue description
Basically, as I understand from standup there are a few obstacles for testing devfiles on prod-preview:

• UI on prod-preview is not up-to-date (should be updated as part of Che 7 rc 4 release)
• I do not think we have enough time to automate tests for all the devfiles in the next 3 weeks
• currently che 7 tests, in general, are not stable enough

@Katka92 could you please provide an update based on the QA involvement section of our Che OSIO ENDGAME plan https://gist.github.com/ibuziuk/cd9eac654daac34ac9d139af988bec01#qa-involvement

[Katka92]

@ibuziuk Me and @ScrewTSW went through the devfiles on prod-preview. The sum up can be found here https://github.com/redhat-developer/che-functional-tests/issues/543.
About current state of tests:

• Once UI is updated, it would require only small changes in tests
• We are focused on automation test for Java Maven now
• The new tests are quite stable, the only issue they are falling on is project import - and I would say it is a problem with Che 7 (today I was able to reproduce that manually, but it happens really rarely).
• Current flow of tests is:
  • Select Java Maven stack
  • Add console-java-simple project
  • Create&Open workspace
  • Wait project is imported
  • Open file
  • Check "Java Language Server" initialisation by suggestion invoking' - PR here

[ibuziuk]

@Katka92 @ScrewTSW thanks for the update - great summary. Hope you will be able to do the second round of testing once we update to rc-4.0

[Katka92]

@ibuziuk We are counting with second round, we believe that the results with rc-4.0 would be much better. 👍

[ibuziuk]

@Katka92 @tdancs it looks like quite a few devfiles from the registry are failing with quota issues e.g. apache-camel / spring-boot. Could you please provide a list of incompatible devfiles in eclipse-che/che#13969

[amisevsk]

Current testing document: https://gist.github.com/amisevsk/fb60ca6bbbcc03f2916a2074fa2ce526

There are a few common issues I've noticed and am working on PRs to address them.

Worth noting however that there are thus far no OSIO-specific issues as far as I can tell.

[ScrewTSW]

[WIP]

01.08.2019
Che version: 7.0.0-rc-4.0
che.prod-preview.OSIO

NodeJS Angular Web Application - OK

• devfile project is automatically selected
• Workspace starts up
• Commands:
  • start - OK
  • build - OK
  • lint - OK
  • yarn install - OK

Apache Camel based on Spring Boot - WARN

• devfile project is automatically selected
• Workspace starts up
• Commands:
  • build the project - OK
  • run the services - WARN - swagger is not working when you try the endpoint request

.NET Core - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • update dependencies - OK
  • run - OK

Go - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • New terminal - opens a new go-cli terminal
  • test outyet - OK
  • run outyet - OK

Java Gradle - WARN

• devfile project is automatically selected
• workspace started
• Commands:
  • gradle build - OK
  • gradle run - OK
  • 8080/tcp - successfully opens up a new tab - THIS IS USELESS, there's no application deployment happening, it's just a single-run simple java hello world.

Java Maven - WARN

• devfile project is automatically selected
• workspace started
• Commands:
  • maven build - OK
  • maven build and run - OK
  • 8080/tcp - successfully opens up a new tab - THIS IS USELESS, there's no application deployment happening, it's just a single-run simple java hello world.

Java with Spring Boot and MySQL - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • mysql
    • prepare database - OK
  • tools
    • maven build - OK
    • run webapp - OK

Java Spring Boot - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • maven build - OK
  • run webapp - OK

Java Vert.x - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • maven build - OK
  • run app - OK

NodeJS Express Web Application - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • run the web app - OK

NodeJS MongoDB Web Application - WARN

• devfile project is automatically selected
• workspace started
• Commands:
  • run the web app - OK
  • create test user - OK
  • nodejs - opening the tab gives Json 404 error

NodeJS React Web Application - FAIL

• devfile project is automatically selected
• workspace started
• Commands:
  • install all required dependencies - OK
  • start the local server - fails to execute command
    No error can be seen in the console logs, the bottom bar only flashes orange for a brief moment

PHP Laravel with MySQL - FAIL

• devfile project is automatically selected
• workspace started
• Commands:
  • Install dependencies - OK
  • Copy the example env file and make the r... - OK
    
    The name of the command is too long and doesn't make sense
  • Generate a new application key - OK
  • Generate a new JWT authentication secr... - OK
    
    Again, the names are too long
  • Start Artisan web server - The application is not accessible after the command run
    
    
  • Stop Artisian web server - OK
  • Run the database migrations - OK

PHP with MySQL - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • Start apace server - OK
  • Configure database - OK
  • Stop apache server - OK
• Example application seems to work fine

PHP with Symfony - FAIL

• devfile project is automatically selected
• workspace started
• Commands:
  • Install dependencies - OK
  • Start Symfony web server - FAIL
    Server started, but the application is unavailable after deplyment
    
    
  • Stop Symfony web server - OK

PHP Simple - FAIL

• devfile project is automatically selected
• workspace started
• Commands:
  • Start Apache web server - FAIL
    After the server is started application is not available
    
  • Stop Apache web server - OK
  • 8080/tcp - command missing, no way to open the deployed webpage in a new tab

Python - OK

• devfile project is automatically selected
• workspace started
• Commands:
  • run - OK

Django - FAIL

• devfile project is automatically selected
• workspace started
• Commands:
  • install dependencies - OK
  • migrate - OK
  • run server - OK
  • django - WARN points to a 404 page
    
  • run server in debug mode - FAIL
    

[amisevsk]

The current state of our devfiles on prod-preview is shown in this gist

There are a few outstanding issues that are upstream issues and prioritized for post-7.0.0:

• The Python extension does not work; this breaks language features and debugging (I think) on python-based devfiles Python plugin randomly crashes eclipse-che/che#14074
• Debugging on most nodejs images doesn't work Debugging nodejs projects seems to be impossible eclipse-che/che#14190
• Running the node mongo sample application is confusing (it's a backend REST API so the preview URL is useless): nodejs-mongo devfile sample issues eclipse-che/che#14009
  • I think it's just a bad sample application

[ibuziuk]

@amisevsk great job! since all the known issues are listed and most of them prioritized for 7.1.0 I would propose to close this issue.
@Katka92 will do verification on QA side as part of the #1490