If I enable Feature Preview the Workload shows "MountVolume.SetUp failed for volume "dev-net-tun" : hostPath type check failed: /dev/net/tun is not a file"

[lucky4ever2]

Describe the bug

I have a Rancher Cluster with RKE.
If I enable Feature Preview the Workload-pod in Rancher shows "MountVolume.SetUp failed for volume "dev-net-tun" : hostPath type check failed: /dev/net/tun is not a file."

The file exists on the node but I think the workspace cannot access it.
I had a similar problem with theia but was able to solve it with the rancher-cluster.yml as follows:

services:
  kubelet:
    extra_binds:
       - /var/gitpod/theia/theia-0.6.0:/var/gitpod/theia/theia-0.6.0:rshared

in this case that doesn't work

Steps to reproduce

Enable Feature Preview in the Gui

Expected behavior

I can use the root access: https://www.gitpod.io/docs/feature-preview/

Additional information

Rancher 2.5.5
RKE 1.2.4
Kubernetes 1.19.6
Gitpod-Selfhost 0.6.0

[csweichel]

Indeed we just assume that file/device is present and available for pods to consume. Once #2657 is sorted, we can test/ensure that the supported environments have this file present. Alternatively, we could let workspaces mknod it itself.

[lucky4ever2]

What can i do?

[rsliotta]

One vote for me too! Same issue.

Centos 7, 5.10 kernel, version 0.6.0 of gitpod self-hosted.

[rsliotta]

Indeed we just assume that file/device is present and available for pods to consume. Once #2657 is sorted, we can test/ensure that the supported environments have this file present. Alternatively, we could let workspaces mknod it itself.

Perhaps the problem is not about /dev/net/tun and more about why is something looking for it? Its looking for it in the base packages too. Maybe something in the gitpod layer itself?

[rsliotta]

Just an additional comment. I checked around to look for solutions and cannot find anything that will fit. This will need to be managed in the solution to set the proper permissions to allow it I believe.

[mehtazubin]

+1
I'm also using a k3s based cluster using Ubuntu 20 LTS. The /dev/net/tun is a character device, not a file. As far as from what I saw, there's no way to configure which the mount type as this is controlled by this line while building workspace pod template: https://github.com/gitpod-io/gitpod/blob/master/components/ws-manager/pkg/manager/create.go#L405

[z3ky]

Just for testing, I deleted the character device and created a simple file /dev/net/tun. Seems that the file can be mounted now, but the subsequent pulling doesn't work

Normal    Scheduled                pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Placed pod [default/ws-45700894-bd6c-4380-a476-b7f5df7776e0] on k8s-master
Normal    Pulling                  pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Pulling image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0"
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Error: ImagePullBackOff
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Error: ErrImagePull
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Failed to pull image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0": rpc error: code = Unknown desc = Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
Normal    BackOff                  pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Back-off pulling image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0"

[rsliotta]

That is a device utilized by docker itself. It's most likely caused by how docker itself is invoked.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[inventionlabsSydney]

Hello!
I can confirm this also exists in Ubuntu 20.04 LTS

root@k8s-workers-2:/home/karl# ls -lah /dev/net/tun
crwxr-xr-x 1 root root 10, 200 Mar 25 00:14 /dev/net/tun
root@k8s-workers-2:/home/karl#

Despite any attempts to work around it - I am fraught with issues including failure in docker pull:

Normal    Scheduled                pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Placed pod [default/ws-89e26e95-e988-40bf-987e-8e8e5c628896] on k8s-worker-24
Normal    Pulling                  pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896  Pulling image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896"
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Error: ImagePullBackOff
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Error: ErrImagePull
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Failed to pull image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896": rpc error: code = Unknown desc = Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
Normal    BackOff                  pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Back-off pulling image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896"

What's also interesting is that the above logs reference an incorrect configuration of the registry endpoint: reg.<subdomain>.<mydomain>.com:3000 which is incorrect, my custom registry is docker-hub.

  imageBuilder:
    registryCerts: []
    registry:
      # name must not end with a "/"
      name: docker.io
      secretName: login-registry-ka-rl-ag
      path: secrets/registry-auth.json
      baseImageName: karlkloppenborg/base-images
      workspaceImageName: karlkloppenborg/workspace-images

This of course only happens when I enable feature preview in the portal.

However once disabling the feature preview, I end up having to delete/install the helm chart to get systems working again because all the workspaces error with "ws-daemon not running" (which it is)

I hope this helps, I'm desperate to get docker-in-docker support running as it's a core part of most of my dev lifecycle :)

Anyways, love your work, also if someone could respond to my email for a enterprise license that'd be great 👍

Thanks,
Karl,

[inventionlabsSydney]

It should also be noted that the only real way around this would be to change the line https://github.com/gitpod-io/gitpod/blob/main/components/ws-manager/pkg/manager/create.go#L336 to point to a reference of

var (
   devCharType          = corev1. HostPathCharDev
)

[geropl]

@rsliotta @lucky4ever2 Currently we have a hard dependecy on Ubuntu 18.04 for workspace nodes (platform support matrix).

@inventionlabsSydney @mehtazubin /dev/net/tun being a char device is the "culprit" here.

@z3ky Your mediaType issue seems unrelated.

I'm closing this for now as we do not plan to support other platforms besides those mentioned in the linked page. If you still want to share progress/ask for help https://community.gitpod.io is a good place to share those.

Please note there is a recent v0.8.0, and we're currently working on v0.9.0.

[inventionlabsSydney]

Hey @geropl! ✋

The platform support matrix states Great or Equal too Ubuntu 18.04 on K3s which is why I decided on Ubuntu 20.04.

I can understand your hesitation to support others however if I was able to provide the required changes and test against 20.04 would this be acceptable?

Thanks,
Karl.

[cyrilcros]

+1 for this