vsix Plugin downloads fail (Ubuntu 20.04/microk8s/chectl)

[mutech]

Describe the bug

Running workspaces fails because of download errors for vsix plugins (independent of the plugin/workspace type).

I installed two instances, one on macos/docker for macos and one on ubuntu using microk8s. The macos installation works fine, the Ubuntu installation consistently fails.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

• Install Ubuntu 20.04 (fresh installation in my case)
• Install microk8s using snap
• Prepare microk8s
  • enable storage, dns, helm
• Install chectl via curl

I tried three installation paths (deleting instance in between installations):

• chectl server:start --installer=helm --platform=microk8s
• chectl server:start --installer=helm --platform=microk8s --multiuser
• chectl server:start --platform=microk8s

All succeed and I can eventually access the dashboard and start a workspace which fails to download plugins after repeated tries:

Processing plugin extension 1/1
  Installing plugin extension 1/1
    Downloading plugin from https://download.jboss.org/...
failed to download plugin from ...

Expected behavior

The workspace starts without failing to download vsix plugins.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: microk8s (v1.19.2)

Screenshots

Installation method

• chectl
  • chectl server:start --platform=microk8s
  • chectl/7.20.0 linux-x64 node-v10.22.1
• OperatorHub
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Eclipse Che Logs

Additional context

I found few issues describing similar effects as a result of mitm ssl-proxies. I'm not using any proxies (not that I know off, I'm a k8s noob, so maybe microk8s is doing stuff I wouldn't expect).

[gattytto]

@mutech

1. can you see an error code for the download fail? like 409 or 404
2. does it happen for the first and all the extension files or only one out of many?
3. are you running ubuntu on the host? you may be running systemd-resolved which in some cases can break dns resolution for pods inside the cluster. if this is the case you may try stopping and disabling systemd-resolved and removing + recreating /etc/hosts (because systemd-resolved replaces it with a symlink).

[mutech]

@gattytto

1. No, didn't find any more specific info
2. Happens to all downloads
3. Will check and update this issue later

Meanwhile I got the workspaces to start up by changing the default dns-server in microk8s (which is using 8.8.8.8/8.8.4.4 by default) to my local dns, so your tip (3) is probably the cause, at least the problem is related to DNS.

Now however I have the problem that from within the workspace, I cannot connect to any web services. Concretely, checking out the demo projects from github fails (theia). When opening a shell wget anything fails (remains in state "connecting to [ip-addr]" - so DNS is working now).

I don't know if this is still in scope of this ticket. Should I open another one?

[gattytto]

@mutech it seems like your issue is not in "che" itself but in the previous layer which is the kubernetes single node cluster. You may want to check about similar issues in microk8s wikis and their gh repo to see if you see any fitting scenario. In the meantime maybe you can start a simple container (like this one) that includes networking tools into your cluster to check out what's going on.

[ericwill]

microk8s support is available but needs to be improved -- there is an issue open for this already: #14455

[mutech]

My bad, it was a firewall issue. Apparently microk8s, even though doing quite a bit of firewall massaging is not by default configuring it to let traffic from pods pass out. Sorry for steering up waves here, but maybe you can drop a hint in your microk8s installation notes to help the next guy stumbling over this: https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s#2-deploying-microk8s