Che deployment on OpenShift should be as simple as on Docker

[l0rd]

The epic is about the few small steps that would make the deployment of Che on OpenShift consistent with the deployment on Docker:

• Add the scripts and doc to run Che on OpenShift Move openshift deployment script and documentation to upstream redhat-developer/rh-che#284
• Adapting default Che runtime stacks to run on OpenShift as unprivileged containers Keep in rh-che Dockerfiles only packages that are not suitable for upstream redhat-developer/rh-che#155
• Don't abort a workspace bootstrap if SSH agent fails to be started (in an unprivileged container) (Che workspaces should boot even if sshd fail to start redhat-developer/rh-che#346)
• Extend Che CLI to start Che on OpenShift (Extend Che CLI to start Che on OpenShift redhat-developer/rh-che#418)
• Inject CHE_* environment variables when bootstrapping Che on OpenShift
• Adapt Docker image eclipse/che-server (alpine based) to run on OpenShift
• Package Che as a minishift addon (c.f. Spike to get Che running on Minishift minishift/minishift#1264 (comment)) (Package Che as a minishift addon redhat-developer/rh-che#353)

[kameshsampath]

@l0rd - how can I issue a PR to this item 5 of this epic ? I have built a script for adding the GITHUB oAuth2

[kameshsampath]

@l0rd , @benoitf - we shall discuss on how to get item 5 in the epic, to be done via cli - as we kind of agreed that using script based approach is no good Ux

[l0rd]

@riuvshin are you working (or worked) on the following subtask?

- [ ] Inject CHE_* environment variables when bootstrapping Che on OpenShift

[riuvshin]

@l0rd no, but I was working on allowing to pass any configuration set to CHE pod without adding new seds to deploy script: #7692 but note that this is done ONLY in che6 branch.

[ibuziuk]

Tests results of running workspaces with stacks available on che6 against minishift v1.9.0+a511b25:

Stack	Workspace starts without errors	Commands for quckstart projects work correctly
.NET	✅	Commands work but for some reason preview URL is not resolved ${server.dot.net.server}
Android	✅	run command error - keytool error: java.io.FileNotFoundException: /home/user/.android/debug.keystore (Permission denied)
Blank	✅	no quickstarts for the stack
C++	✅	✅
CentOS blank	✅	no quickstarts for the stack
CentOS Go	✅	✅
CentOS Node.js	✅	Unable to open /etc/scl/conf/rh-nodejs4! and quickstart itself seems to have issues with dependencies bower install - ECONFLICT Unable to find suitable version for jquery
CentOS Wildfly Swarm	✅	wfswarm-rest-http quickstart - build command fails [1]
Ceylon with Java JavaScript Dart on CentOS	🔴 [2]	❓
Debian	🔴 [2]	❓
Debian LSP	🔴 [2]	❓
Eclipse Che	✅	build command stucks at [INFO] Compiling module org.eclipse.che.ide.IDE
Eclipse Vert.x	✅	vertx-http-booster' build command fails [3]
Go	✅	✅
Hadoop	✅	✅
Java	✅	✅ web-java-spring sample works just fine!
Java CentOS	✅	web-java-spring sample command web-java-spring:build - cp: cannot create regular file '/home/user/tomcat8/webapps/ROOT.war': Permission denied / SEVERE: Cannot start server. Server instance is not configured.
Java Debian	✅	web-java-spring sample command web-java-spring:build - cp: cannot create regular file '/home/user/tomcat8/webapps/ROOT.war': Permission denied / SEVERE: Cannot start server. Server instance is not configured.
Java MySql	Infrastructure not found for type: compose - "Compose" is not supported by OpenShift infrastructure - failure is expected
Java MySql CentOS	Infrastructure not found for type: compose - "Compose" is not supported by OpenShift infrastructure - failure is expected
Kotlin	✅	no quickstarts for the stack
Node	✅	✅
Openshift Default	✅	no quickstarts for the stack
OpenShift Sql	✅	no quickstarts for the stack
PHP	🔴 [4]	❓
PHP GAE	🔴 [4]	❓
PHP-5.6	✅	sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted
PlatformIO	✅	``pio account login command and try again. / Error: Not a PlatformIO project. `platformio.ini` file has not been found in current working directory (/projects/.chexq86iy). To initialize new project please use `platformio init` command`
Python	✅	Python language initialization failure [6]
Python 2.7	✅	Python language initialization failure [6] PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted [5]
Python GAE	✅	Python language initialization failure [6]
Rails	🔴 - can not pull the image [7] and permission denied issue	❓
Selenium	✅	no quickstarts for the stack
Spring Boot	✅	spring-boot-http-booster build command fails [8]
TomEE	🔴 [10]	❓
Ubuntu	🔴 [9]	no quickstarts for the stack
Zend	🔴 [10]	❓

[1] [WARNING] Failed to create parent directories for tracking file /home/user/.m2/repository/io/openshift/booster-parent/10/booster-parent-10.pom.lastUpdated [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for io.openshift.booster:wfswarm-rest-http:11-SNAPSHOT: Could not transfer artifact io.openshift:booster-parent:pom:10 from/to central (https://repo1.maven.org/maven2): /home/user/.

[2] mkdir: cannot create directory '/home/user': Permission denied Exec Agent binary is downloaded remotely gzip: stdin: unexpected end of file tar: Error is not recoverable: exiting now tar: Child returned status 1 /bin/bash: line 227: /home/user/che/exec-agent/che-exec-agent: Permission denied)

[3] scl enable rh-maven33 'mvn clean install -f /projects/vertx-http-booster' [INFO] Scanning for projects... Downloading: https://repo1.maven.org/maven2/io/openshift/booster-parent/13/booster-parent-13.pom [WARNING] Failed to create parent directories for tracking file /home/user/.m2/repository/io/openshift/booster-parent/13/booster-parent-13.pom.lastUpdated [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for io.openshift.booster:http-vertx:18-SNAPSHOT: Could not transfer artifact io.openshift:booster-parent:pom:13 from/to central (https://repo1.maven.org/maven2): /home/user/.m2/repository/io/openshift/booster-parent/13/booster-parent-13.pom.part.lock (No such file or directory) and 'parent.relativePath' points at wrong local POM @ line 5, column 11
NOTE image registry.centos.org/che-stacks/vertx

[4] 2017-12-12 15:15:15,728[default.svc/...] [ERROR] [.k.c.d.i.ExecWebSocketListener 213] - Exec Failure: HTTP:500. Message:container not found ("container")   | java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error'   | at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)   | at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)   | at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)   | at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)   | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)   | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)   | at java.lang.Thread.run(Thread.java:748)

[5] sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted sudo: no valid sudoers sources found, quitting sudo: setresuid() [0, 0, 0] -> [1000420000, -1, -1]: Operation not permitted sudo: unable to initialize policy plugin sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted sudo: no valid sudoers sources found, quitting sudo: setresuid() [0, 0, 0] -> [1000420000, -1, -1]: Operation not permitted sudo: unable to initialize policy plugin sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted sudo: no valid sudoers sources found, quitting sudo: setresuid() [0, 0, 0] -> [1000420000, -1, -1]: Operation not permitted sudo: unable to initialize policy plugin Traceback (most recent call last): File "main.py", line 1, in <module> from flask import Flask ImportError: No module named flask

[6]

[7] docker pull eclipse/ubuntu_rails Using default tag: latest Error response from daemon: Get https://registry-1.docker.io/v2/eclipse/ubuntu_rails/manifests/latest: Get https://auth.docker.io/token?account=ibuziuk&scope=repository%3Aeclipse%2Fubuntu_rails%3Apull&service=registry.docker.io: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Also Permission denied problem:
`mkdir: cannot create directory ‘//che’: Permission denied
Exec Agent binary is downloaded remotely

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
/bin/bash: line 227: //che/exec-agent/che-exec-agent: No such file or directory`

[8] [WARNING] Failed to create parent directories for tracking file /home/user/.m2/repository/io/openshift/booster-parent/11/booster-parent-11.pom.lastUpdated [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for io.openshift.booster:spring-boot-rest-http:14-SNAPSHOT: Could not transfer artifact io.openshift:booster-parent:pom:11 from/to central (https://repo1.maven.org/maven2): /home/user/.m2/repository/io/openshift/booster-parent/11/booster-parent-11.pom.part.lock (No such file or directory) and 'parent.relativePath' points at wrong local POM @ line 20, column 11 [WARNING] Failed to create parent directories for tracking file /home/user/.m2/repository/io/openshift/booster-parent/11/booster-parent-11.pom.lastUpdated [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for io.openshift.booster:spring-boot-rest-http:14-SNAPSHOT: Could not transfer artifact io.openshift:booster-parent:pom:11 from/to central (https://repo1.maven.org/maven2): /home/user/.m2/repository/io/openshift/booster-parent/11/booster-parent-11.pom.part.lock (No such file or directory) and 'parent.relativePath' points at wrong local POM @ line 20, column 11

[9] mkdir: cannot create directory ‘//che’: Permission denied Exec Agent binary is downloaded remotely gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now /bin/bash: line 227: //che/exec-agent/che-exec-agent: No such file or directory

[10] 2017-12-12 16:07:00,200[aceSharedPool-0] [WARN ] [w.i.o.OpenShiftInternalRuntime 131] - Failed to start OpenShift runtime of workspace workspacexhgkyzmrlioctvf7. Cause: Waiting for pod 'workspacexhgkyzmrlioctvf7.dockerimage' reached timeout  | 2017-12-12 16:07:00,442[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 278] - Workspace 'che:wksp-wjab' with id 'workspacexhgkyzmrlioctvf7' start failed - for some reason workspace pod is not started - no errors in OpenShift events

[ibuziuk]

@sleshchenko @akorneta do you guys have any ideas about why centos based stacks from che6 are failing on minishift ?

[1] #6097 (comment)

[ghost]

@ibuziuk what exact stacks?

[ibuziuk]

@eivantsov please, take a look at the table in the comment above

[sleshchenko]

@ibuziuk For me looks like the minishift is not a cause of failure and it will work in the same way for OCP and OSIO.
2, 5, 6, 7, 9 and maybe some of the other issues fail because of lack of root permissions. It is needed to investigate how to solve this issue. Are they eclipse images? If not, we can create our ones and pre-create required folders or set required permissions.
10 - maybe entrypoint has short living process while our workspaces require endless like (tail -f /dev/null)

[ghost]

I know why php images fail (sudo in cmd) and Centos and Debian ones (permissions for user home dir) too. That should be an easy fix.

Others, including sample build failures and language server initialization errors, should be investigated as separate issues.

@sleshchenko yes, these are eclipse images that we control

[garagatyi]

5c from me:
Zend stack uses sudo in CMD and fails because of that. https://hub.docker.com/r/kaloyanraev/che-zendserver/~/dockerfile/

[ghost]

@ibuziuk Centos based stacks are smth that your team builds. I have no control over them.

For example, this image registry.centos.org/che-stacks/centos-stack-base:latest does not see to have smth that this one has https://github.com/eclipse/che-dockerfiles/blob/master/recipes/stack-base/centos/Dockerfile#L45

Why are we using this registry, not DockerHub and Ecipse account?

[l0rd]

@ibuziuk are these problems related to che6 (i.e. no error when running these runtime stacks on che5?)

@eivantsov @sleshchenko I think @ibuziuk was only asking if you had an idea about the failure. Fixing these stacks is one of the Che 6 tasks that che-osio team should work on (task n.2 of #6097).

[ghost]

@l0rd I'm fixing php stacks now. Got a way to run apache on port 80 without sudo.

As to CentOS stacks, please make sure the base image has this https://github.com/eclipse/che-dockerfiles/blob/master/recipes/stack-base/centos/Dockerfile#L45

[l0rd]

@l0rd I'm fixing php stacks now. Got a way to run apache on port 80 without sudo.

ok cool

As to CentOS stacks, please make sure the base image has this https://github.com/eclipse/che-dockerfiles/blob/master/recipes/stack-base/centos/Dockerfile#L45

Yes will send an email to dharmit and copying you too so you get familiar with this process.

[ibuziuk]

@l0rd not sure if it is only che6 related (have not done this research against che5 - need to check which images are used there)
@eivantsov thanks for looking at php stacks ;-)

Why are we using this registry, not DockerHub and Ecipse account?
For osio we were forced to use this registry due to security concerns AFAIK, but in che it would be more consistent to have all the images under Eclipse hood on Dockerhub @l0rd WDYT ?

[ghost]

@ibuziuk @l0rd it is not Che6 related. Most issues are permissions related.

[l0rd]

@ibuziuk I think that centos container pipeline is a better source for centos based images. The only concern I have with the centos container pipeline is that we don't control the build (e.g. if the build of an image fail we are not notified) and that's something we need to address.

[l0rd]

#7967 should fix Ceylon, Debian, DebianLSP, Ubuntu and Ruby on Rails stacks (workspace start without errors).

I have pinged @dharmit for the centOS stacks. It looks like the images are ok but there is a discrepancy between the stacks defined in the index and the ones that appears in the centos registry UI.

@eivantsov should we fix Zend and TomEE too? I can create a PR to fix them if it's worth.

@skryzhny is there an issue for the support of Java MySQL and Java MySQL CentOS stacks (type compose) on OpenShift?

[ghost]

@l0rd I looked at Zend briefly.. a lost of sudo invoked by the original command. Moreover, it's not eclipse images that is used there

[sleshchenko]

@l0rd Yes, here is an issue for supporting compose recipes by OpenShift infra #7860

[l0rd]

I am checking the following subtask as complete because all stacks that we maintain can start successfully on OpenShift (except compose based ones):

• Adapting default Che runtime stacks to run on OpenShift as unprivileged containers Keep in rh-che Dockerfiles only packages that are not suitable for upstream redhat-developer/rh-che#155

I have opened three new issues for 1) the quickstarts that don't work #7990 2) the Zend stack that doesn't start #7989 3) and the TomEE stacks that doesn't start #7991

[l0rd]

Closing this issue since the only task left was about updating Che CLI and we decided to not implement it and making the minishift addon as the default way to start Che on minishift (redhat-developer/rh-che#506).