If I enable Feature Preview the Workload shows "MountVolume.SetUp failed for volume "dev-net-tun" : hostPath type check failed: /dev/net/tun is not a file"

[lucky4ever2]

Describe the bug

I have a Rancher Cluster with RKE.
If I enable Feature Preview the Workload-pod in Rancher shows "MountVolume.SetUp failed for volume "dev-net-tun" : hostPath type check failed: /dev/net/tun is not a file."

The file exists on the node but I think the workspace cannot access it.
I had a similar problem with theia but was able to solve it with the rancher-cluster.yml as follows:

services:
  kubelet:
    extra_binds:
       - /var/gitpod/theia/theia-0.6.0:/var/gitpod/theia/theia-0.6.0:rshared

in this case that doesn't work

Steps to reproduce

Enable Feature Preview in the Gui

Expected behavior

I can use the root access: https://www.gitpod.io/docs/feature-preview/

Additional information

Rancher 2.5.5
RKE 1.2.4
Kubernetes 1.19.6
Gitpod-Selfhost 0.6.0

[csweichel]

Indeed we just assume that file/device is present and available for pods to consume. Once #2657 is sorted, we can test/ensure that the supported environments have this file present. Alternatively, we could let workspaces mknod it itself.

[lucky4ever2]

What can i do?

[rsliotta]

One vote for me too! Same issue.

Centos 7, 5.10 kernel, version 0.6.0 of gitpod self-hosted.

[rsliotta]

Indeed we just assume that file/device is present and available for pods to consume. Once #2657 is sorted, we can test/ensure that the supported environments have this file present. Alternatively, we could let workspaces mknod it itself.

Perhaps the problem is not about /dev/net/tun and more about why is something looking for it? Its looking for it in the base packages too. Maybe something in the gitpod layer itself?

[rsliotta]

Just an additional comment. I checked around to look for solutions and cannot find anything that will fit. This will need to be managed in the solution to set the proper permissions to allow it I believe.

[mehtazubin]

+1
I'm also using a k3s based cluster using Ubuntu 20 LTS. The /dev/net/tun is a character device, not a file. As far as from what I saw, there's no way to configure which the mount type as this is controlled by this line while building workspace pod template: https://github.com/gitpod-io/gitpod/blob/master/components/ws-manager/pkg/manager/create.go#L405

[z3ky]

Just for testing, I deleted the character device and created a simple file /dev/net/tun. Seems that the file can be mounted now, but the subsequent pulling doesn't work

Normal    Scheduled                pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Placed pod [default/ws-45700894-bd6c-4380-a476-b7f5df7776e0] on k8s-master
Normal    Pulling                  pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Pulling image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0"
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Error: ImagePullBackOff
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Error: ErrImagePull
Warning   Failed                   pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Failed to pull image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0": rpc error: code = Unknown desc = Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
Normal    BackOff                  pod/ws-45700894-bd6c-4380-a476-b7f5df7776e0   Back-off pulling image "reg.mydomain.com:3000/remote/45700894-bd6c-4380-a476-b7f5df7776e0"

[rsliotta]

That is a device utilized by docker itself. It's most likely caused by how docker itself is invoked.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[inventionlabsSydney]

Hello!
I can confirm this also exists in Ubuntu 20.04 LTS

root@k8s-workers-2:/home/karl# ls -lah /dev/net/tun
crwxr-xr-x 1 root root 10, 200 Mar 25 00:14 /dev/net/tun
root@k8s-workers-2:/home/karl#

Despite any attempts to work around it - I am fraught with issues including failure in docker pull:

Normal    Scheduled                pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Placed pod [default/ws-89e26e95-e988-40bf-987e-8e8e5c628896] on k8s-worker-24
Normal    Pulling                  pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896  Pulling image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896"
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Error: ImagePullBackOff
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Error: ErrImagePull
Warning   Failed                   pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Failed to pull image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896": rpc error: code = Unknown desc = Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
Normal    BackOff                  pod/ws-89e26e95-e988-40bf-987e-8e8e5c628896   Back-off pulling image "reg.<subdomain>.<mydomain>.com:3000/remote/89e26e95-e988-40bf-987e-8e8e5c628896"

What's also interesting is that the above logs reference an incorrect configuration of the registry endpoint: reg.<subdomain>.<mydomain>.com:3000 which is incorrect, my custom registry is docker-hub.

  imageBuilder:
    registryCerts: []
    registry:
      # name must not end with a "/"
      name: docker.io
      secretName: login-registry-ka-rl-ag
      path: secrets/registry-auth.json
      baseImageName: karlkloppenborg/base-images
      workspaceImageName: karlkloppenborg/workspace-images

This of course only happens when I enable feature preview in the portal.

However once disabling the feature preview, I end up having to delete/install the helm chart to get systems working again because all the workspaces error with "ws-daemon not running" (which it is)

I hope this helps, I'm desperate to get docker-in-docker support running as it's a core part of most of my dev lifecycle :)

Anyways, love your work, also if someone could respond to my email for a enterprise license that'd be great 👍

Thanks,
Karl,