permission denied while creating volume path

[benoitf]

Describe the bug

Workspace is not starting and report "timeout"
In workspace pod events we can see

Error: Error response from daemon: create b72f86e6ddffd62eaaa51d33db4b993d11c88396309d735678899fddfc10ab2f: error while creating volume path '/var/lib/docker/volumes/b72f86e6ddffd62eaaa51d33db4b993d11c88396309d735678899fddfc10ab2f/_data': mkdir /var/lib/docker/volumes/b72f86e6ddffd62eaaa51d33db4b993d11c88396309d735678899fddfc10ab2f: permission denied

Here is the dockerfile of the container image used:

FROM centos:8
RUN mkdir -p /foo && chmod 777 /foo && chgrp 0 /foo; chmod g+rwX /foo
VOLUME /foo
CMD tail -f /dev/null

The main error is coming with the VOLUME instruction

Che version

• latest
• nightly
• other: please specify : che.openshift.io

Steps to reproduce

https://che.openshift.io/dashboard/#/load-factory?url=https://gist.githubusercontent.com/benoitf/b1ef68e9c9daece33e341a0cd0b1b046/raw/badf0b47da7dd448e80ed5ab8000bd909dc6f6b8/gistfile1.txt

Expected behavior

Workspace is able to start

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: che.openshift.io

Screenshots

[azatsarynnyy]

I guess it's from wsmater area. It looks like a P1.

[skabashnyuk]

@benoitf is this reproduced on minikube/minishift or is this have some hosted che specific?
CC @ibuziuk

[amisevsk]

This is an OpenShift issue -- any container that has a VOLUME directive in its dockerfile has to have a corresponding volumeMount.

The solution is to add a volume to the devfile:

apiVersion: 1.0.0

metadata:
  name: fbenoit-test1
  
projects:
  - name: python-hello-world
    source:
      type: git
      location: 'https://github.com/datawire/hello-world-python.git'
components:
  - id: ms-python/python/latest
    type: chePlugin
  - type: dockerimage
    image: docker.io/florentbenoit/test3
    memoryLimit: 512Mi
    volumes:
      - name: foo
        containerPath: /foo

There's not a whole lot we can do on the Che side automatically here. Without a volumeMount, there's no good way to determine where to allocate the storage for the container, and without the dockerfile there's no way to know which volumes we need to add.

[l0rd]

This is an OpenShift issue -- any container that has a VOLUME directive in its dockerfile has to have a corresponding volumeMount.

@amisevsk are you sure about that?

[metlos]

We've been hit by this ourselves when the volume mounts for jwt-proxy container strayed away from the VOLUME def in che-jwt-proxy docker image.

[amisevsk]

@l0rd It's probably a configuration setting in the cluster itself (not that I can find documentation about it), but without a volumemount, 3.x clusters will try to create a volume in /var/lib/docker/volumes; on every non-minishift cluster I've used this has been forbidden -- I'd at least consider it best-practices in the general case. We've had to work around this at various times in the past, e.g. eclipse-che/che-devfile-registry#14, eclipse-che/che-devfile-registry#188, redhat-developer/kubernetes-image-puller#3 (comment).

Edit: found some docs for 3.11 at least: https://docs.openshift.com/container-platform/3.11/install/host_preparation.html#blocking-local-volume-usage

Edit2: Additional info: bugzilla, KB article. Looks like CRI-O can handle this gracefully, but it's still a cluster/host config issue, not a Che issue.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.