Cannot start workspace with plugins on a Che local instance in Minishift

[apupier]

Description

i cannot start a workspace with a plugin activated. there si always this error:

exit status 1
Error: Failed to run the workspace: "Plugins installation process failed. Error: Plugin broking process for workspace workspaceoa628vydrzj2s390 failed with error: exit status 1"

Reproduction Steps

• Create a workspace based on a Che 7 stack
• Go to workspaces to configure it to add a Plugin (i tried with java, XML and Python)
• try to restart the workspace

OS and version:

Windows 10
installed on minishift using CDK 3.8.0
the Che version pulled is 7.0.0-beta-3.0

Diagnostics:
if i go to the che-plugin-broker pod when the workspace is trying to start, there is this kind of log:

2019/04/12 14:05:37 Broker configuration
2019/04/12 14:05:37   Push endpoint: ws://che-mini-che.172.18.93.39.nip.io/api/websocket
2019/04/12 14:05:37   Auth enabled: false
2019/04/12 14:05:37   Runtime ID:
2019/04/12 14:05:37     Workspace: workspaceoa628vydrzj2s390
2019/04/12 14:05:37     Environment: default
2019/04/12 14:05:37     OwnerId: che
2019/04/12 14:05:37 Unified Che Plugin Broker
2019/04/12 14:05:37 List of plugins and editors to install
- che-machine-exec-plugin:0.0.1 - Che Plug-in with che-machine-exec service to provide creation terminal or tasks for Eclipse CHE workspace machines.
- ms-python.python:2019.2.5433 - Linting, Debugging (multi-threaded, remote), Intellisense, code formatting, refactoring, unit tests, snippets, and more.
- org.eclipse.che.editor.theia:next - Eclipse Theia, get the latest release each day.
2019/04/12 14:05:37 Starting Che common plugins processing
2019/04/12 14:05:37 Stared processing plugin 'che-machine-exec-plugin:0.0.1'
2019/04/12 14:05:37 Downloading archive 'https://github.com/eclipse/che-machine-exec/releases/download/0.0.2/che-service-plugin.tar.gz'  for plugin 'che-machine-exec-plugin:0.0.1' to '/tmp/che-plugin-broker443678868/pluginArchive.tar.gz'
2019/04/12 14:05:38 Untarring archive 'https://github.com/eclipse/che-machine-exec/releases/download/0.0.2/che-service-plugin.tar.gz'  for plugin 'che-machine-exec-plugin:0.0.1' to '/tmp/che-plugin-broker443678868/pluginArchive.tar.gz'
2019/04/12 14:05:38 Resolving 'che-machine-exec-plugin:0.0.1'
2019/04/12 14:05:38 Copying dependencies for 'che-machine-exec-plugin:0.0.1'
2019/04/12 14:05:38 Stared processing plugin 'org.eclipse.che.editor.theia:next'
2019/04/12 14:05:38 Downloading archive 'https://github.com/eclipse/che-theia/releases/download/next/che-editor-plugin.tar.gz'  for plugin 'org.eclipse.che.editor.theia:next' to '/tmp/che-plugin-broker250082019/pluginArchive.tar.gz'
2019/04/12 14:05:38 Untarring archive 'https://github.com/eclipse/che-theia/releases/download/next/che-editor-plugin.tar.gz'  for plugin 'org.eclipse.che.editor.theia:next' to '/tmp/che-plugin-broker250082019/pluginArchive.tar.gz'
2019/04/12 14:05:38 Resolving 'org.eclipse.che.editor.theia:next'
2019/04/12 14:05:38 Copying dependencies for 'org.eclipse.che.editor.theia:next'
2019/04/12 14:05:38 Starting Theia plugins processing
2019/04/12 14:05:38 Starting VS Code plugins processing
2019/04/12 14:05:38 Stared processing plugin 'ms-python.python:2019.2.5433'
2019/04/12 14:05:38 Downloading VS Code extension archive 'https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-python/vsextensions/python/2019.2.5433/vspackage'  for plugin 'ms-python.python:2019.2.5433' to '/tmp/vscode-extension-broker556251878/pluginArchivelhexeqxexf'
2019/04/12 14:05:38 Downloading VS Code extension for plugin 'ms-python.python:2019.2.5433'
2019/04/12 14:05:43 Unzipping archive '/tmp/vscode-extension-broker556251878/pluginArchivelhexeqxexf' for plugin 'ms-python.python:2019.2.5433' to '/tmp/vscode-extension-broker556251878/extension/mkekwpljme'
2019/04/12 14:05:46 Copying VS Code extension 'ms-python.python:2019.2.5433' from '/tmp/vscode-extension-broker556251878/extension/mkekwpljme' to '/plugins/ms-python.python.2019.2.5433.mspython_python'
2019/04/12 14:05:46 exit status 1

In events of the pods:

(the only strange thing is that there is a che-init-plugin-broker which is more recent on docker hub and with a version corresponding to the che-unified-plugin-broker
v0.15.1 available although version 0.14.0 is used https://hub.docker.com/r/eclipse/che-init-plugin-broker/tags )

[apupier]

for information, I used Che minishift addon to do the installation in case it can be the cause for the troubles

[amisevsk]

@apupier Sorry to let this issue languish for so long. A fair bit has changed over the last few weeks, could you try to reproduce the issue with:

• image: che-server:7.0.0-beta-4.0
• ensure CHE_WORKSPACE_PLUGIN__REGISTRY__URL ends in v2

Both of the above should be done by updating the minishift addon and re-applying.

[themr0c]

Hi, I have a similar issue starting a Che7 workspace on minishift v1.34.0+f5db7cb.

Che is installed as minishift addon.

In minishift, at project creation I specified:

• Eclipse Che version = 7.0.0-beta-5.0
• Deploy Che in a debug mode = true
• CHE_WORKSPACE_PLUGIN__REGISTRY__URL I tried both unset or set to https://che-plugin-registry.openshift.io/v3

I get following error in Che:

Error: Failed to run the workspace: "Plugins installation process timed out"

In minishift, I get following (and at the moment I didn't found a way to get a more verbose error output)

Init container eclipse-che-init-plugin-broker-v0-17
State: Terminated at May 29, 2019 12:25:52 PM with exit code 1 (Error)

Init Containers
eclipse-che-init-plugin-broker-v0-17
Image: eclipse/che-init-plugin-broker:v0.17
Command: <image-entrypoint> -push-endpoint ws://che-7-beta-5./api/websocket -runtime-id workspaceqvxo44iqvndzhfq8:default:che --registry-address https://che-plugin-registry.openshift.io/v3
Collapse
Mount: claim-che-workspacezlxvzqxq, subpath workspaceqvxo44iqvndzhfq8/plugins/ → /plugins read-write
Mount: claim-che-workspaceglpw8hti, subpath workspaceqvxo44iqvndzhfq8/che-logs-che-plugin-broker/ → /workspace_logs read-write
Mount: default-token-pv6tz → /var/run/secrets/kubernetes.io/serviceaccount read-only
Memory: 250 MiB to 250 MiB 

[amisevsk]

@themr0c Hm, that is strange. A few clarifying questions:

• When starting a workspace, is it the init container (eclipse-che-init-plugin-broker) or main container (eclipse-che-unified-plugin-broker) that fails?
• Which plugins does the workspace contain?
• If you are able to get the broker logs, it would be helpful; from the openshift console (execute minishift console), you should see a pod named something like che-plugin-broker from which you should be able to get logs, but it only sticks around while the workspace is starting so you have to be kind of quick.

[themr0c]

@amisevsk my issue was related to the last changes in API (v1 then v2 and v3). See #13439 and the related PR. I managed to get things working with minishift.

[nickboldt]

This is still occurring with minishift v1.33.0+ba29431b, which pulls these images:

$➔ docker images | grep che
eclipse/che-remote-plugin-runner-java11                         next                f24ecc40cd2b        7 hours ago         770MB
eclipse/che-theia                                               next                18e5ca060d25        7 hours ago         274MB
eclipse/che-server                                              latest              982a0d6326cb        11 days ago         361MB
eclipse/che-machine-exec                                        latest              2279c8d5a39b        2 weeks ago         25.5MB
eclipse/che-unified-plugin-broker                               v0.17               60efc0fab2bc        2 weeks ago         12.2MB
eclipse/che-init-plugin-broker                                  v0.17               dfe37a713acc        4 weeks ago         5.78MB
eclipse/che-dev                                                 7.0.0-RC-1.0        22137654a486        6 weeks ago         1.63GB

When I start a Che workspace, this happens:

Starting Init Plugin Broker
Cleaning /plugins dir
Failed to download plugin meta: failed to fetch plugin meta.yaml for 
plugin 'eclipse/che-machine-exec-plugin/0.0.1' from registry 
'https://che-plugin-registry.openshift.io/plugins': 
Downloading https://che-plugin-registry.openshift.io/plugins/eclipse/
che-machine-exec-plugin/0.0.1/meta.yaml failed. Status code 404

Workaround:

1. Go to your minishift's mini-che deployment's Deployment Config, eg.:

https://192.168.99.111:8443/console/project/mini-che/edit/dc/che

2. Scroll down and look for CHE_WORKSPACE_PLUGIN__REGISTRY__URL

3. Set it to https://che-plugin-registry.openshift.io/v3, then save.

4. Wait until Che dashboard is redeployed. Now you can run Che 7 RC1:

[nickboldt]

And I see this is fixed upstream in minishift >1.34.0:
minishift/minishift#3273 -> minishift/minishift@v1.34.0...master

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.