Volume sharing between init container and che-plugin container does not working

[mshaposhnik]

Describe the bug

Volume sharing between init container and che-plugin container seems does not working.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

Consider the following devfile, which fills some volume in init container, and want to read it contents
in chePlugin type component container:

devfile

metadata:
  name: java-maven-m5p8u
projects:
  - name: console-java-simple
    source:
      location: 'https://github.com/che-samples/console-java-simple.git'
      type: git
      branch: java1.11
attributes:
  persistVolumes: 'false'
components:
  - id: redhat/java11/latest
    type: chePlugin
  - referenceContent: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: ws
      spec:
        initContainers:
          - name: init-myservice
            image: adoptopenjdk:11-jre-openj9
            command: ['sh', '-c', "echo starting && cp -R /opt/java/openjdk/ /tmp/jdk && echo done && sleep 20"]
            volumeMounts: 
              - name: jdk
                mountPath: /tmp/jdk
        volumes:
          - name: jdk
            emtpyDir: {}
    type: kubernetes
  - mountSources: true
    memoryLimit: 512Mi
    type: dockerimage
    volumes:
      - name: jdk
        containerPath: /tmp/jdk
      - name: m2
        containerPath: /home/user/.m2
    image: 'quay.io/eclipse/che-java11-maven:7.13.1'
    alias: maven
    env:
      - value: ''
        name: MAVEN_CONFIG
      - value: >-
          -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
          -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true
          -Xms20m -Djava.security.egd=file:/dev/./urandom
        name: JAVA_OPTS
      - value: >-
          -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
          -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true
          -Xms20m -Djava.security.egd=file:/dev/./urandom -Duser.home=/home/user
        name: MAVEN_OPTS
      - value: >-
          -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
          -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
          -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true
          -Xms20m -Djava.security.egd=file:/dev/./urandom
        name: JAVA_TOOL_OPTIONS
apiVersion: 1.0.0
commands:
  - name: maven build
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/console-java-simple'
        type: exec
        command: mvn clean install
        component: maven
  - name: maven build and run
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/console-java-simple'
        type: exec
        command: mvn clean install && java -jar ./target/*.jar
        component: maven
</details>

Expected behavior

It's expected to see volule non-empty.

[mshaposhnik]

Looking into workspace resulting yaml, here is mounts:

1. Init container:

volumeMounts:
   - mountPath: /tmp/jdk
     name: jdk

2. maven component

volumeMounts:
    - mountPath: /tmp/jdk
      name: claim-che-workspacebh18r787
      subPath: workspace3svcq0fm84lxaqbn/jdk/

Pod Volumes (except secret):

 volumes:
  - configMap:
      defaultMode: 420
      name: workspace3svcq0fm84lxaqbn.jwtproxy-config
    name: che-jwtproxy-config-volume
  - emptyDir: {}
    name: jdk
  - emptyDir: {}
    name: m2
  - emptyDir: {}
    name: plugins
  - emptyDir: {}
    name: remote-endpoint
  - configMap:
      defaultMode: 420
      name: workspace3svcq0fm84lxaqbn.broker-config-mapyrbyh0
    name: broker-config-volume5i7lkh
  - emptyDir: {}
    name: claim-che-workspacebh18r787
  - emptyDir: {}
    name: claim-che-workspacewslz2xk2
  - emptyDir: {}
    name: claim-che-workspacemnvkymgh

So seems mount in init and plugin containers refers to different volumes of the pod.

[ericwill]

This issue blocks #17924

[nickboldt]

Per Eric's comments, setting milestone 7.21 for now

[mshaposhnik]

@ericwill Can you please concretize a bit your exact usecase. I mean, how do you define init containers and volumes inside of it (is is an K8S component with volumes defined at it's recipe level or the volume is at the devile component level etc), what kind of dependable component etc.
Why i'm asking is because the problem seems may have different variations, and, therefore, fixing approaches, so i want to first look into the particular problem branch which will unblock your work.

[tsmaeder]

@mshaposhnik as described in #16851 (comment), we're trying to copy a directory from a container to an ephemeral volume so that we can reuse that directory in a plugin component. We don't really care what component that init container is attached to.
What we would really like to do is mount a directory from one container (open J9) in a different container (vscode-java), but that seems impossible in Kubernetes.

[mshaposhnik]

@ericwill @tsmaeder we have a working solution prototype, but it is should be configured a bit differently.
Because we're propogating Kubernetes component recipe directly to workspace deployment, and that cannot be easily changed. So our fix will allow to init containers use a cross-component shared volumes by defining those volumes at a devfile component level, e.g. the devfile will looks like:

components:
  - id: redhat/java11/latest
    type: chePlugin
  - referenceContent: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: ws
      spec:
        initContainers:
          - name: init-myservice
            image: adoptopenjdk:11-jre-openj9
            command: ['sh', '-c', "echo starting && cp -R /opt/java/openjdk/ /tmp/jdk && echo done && sleep 20"]
-->>        # no volumes and/or mounts inside the init container recipe. they will be provisioned from component volume
    volumes: 
      - name: jdk
        containerPath: /tmp/jdk  # refers the same volume as maven component below
    type: kubernetes
  - mountSources: true
    memoryLimit: 512Mi
    type: dockerimage
    volumes:
      - name: jdk
        containerPath: /tmp/jdk  

cc @skabashnyuk @metlos

[tsmaeder]

@mshaposhnik will this create an ephemeral volume? That's the important part.

[mshaposhnik]

@tsmaeder yep, nothing changed there. We just add provisioing of the devfile volumes into init containers.

[tsmaeder]

I just don't see how we can tell when to allocate a persistent volume and when to allocate local storage.

[mshaposhnik]

what do you mean by "local storage"? Whole workspace storage can be persistent or ephemeral. it can't be specified per-component.

[tsmaeder]

@mshaposhnik if you read #16851 (comment) and later comments, it should become clear what the requirements are. Persistent volumes are 10x slower (at least on openshift.io) and therefore no good.

[sleshchenko]

what do you mean by "local storage"? Whole workspace storage can be persistent or ephemeral. it can't be specified per-component.

It can ) Plugin metadata allows requiring ephemeral persistence independent from workspace storage type. https://github.com/eclipse/che-plugin-registry/blob/master/v3/plugins/eclipse/che-theia/next/meta.yaml#L88 ( the PR where it was implemented on Che Server side #14539 )
But we identify volumes via name, it means that if different components ask for volume with the same name but different types - workspace should fail to start with the corresponding message.
Maybe we should introduce the same volume attribute on the devfile 1.x format.

[mshaposhnik]

Ok it can be different for particular che-plugin, but that's imo little out of the current context (IMO this check should have been implemented in the original PR). The main question for now is how to deal with volumes described directly in K8S/OS recipes and align them with other workspace volumes in terms of sharing, type etc...

[mshaposhnik]

Fix is merged now.

[tsmaeder]

@mshaposhnik do we have reference documentation that shows the different ways I can define init containers in a devfile? Is there a way to define an init container for the whole workspace?

[tsmaeder]

When I add the "ephemeral" field to a volume I define in the devfile, che nightly complains that the devfile is not valid.

[tsmaeder]

  - mountSources: true
    memoryLimit: 512Mi
    type: dockerimage
    volumes:
      - name: jdk
        containerPath: /tmp/jdk
        ephemeral: true
      - name: m2
        containerPath: /home/user/.m2
    image: 'quay.io/eclipse/che-java11-maven:7.13.1'
    alias: maven
    env:
    ....

[tsmaeder]

The full monty:

apiVersion: 1.0.0
metadata:
  name: testJDKVersionKube
attributes:
  persistVolumes: 'true'
projects:
  - name: console-java-simple
    source:
      location: 'https://github.com/che-samples/console-java-simple.git'
      type: git
      branch: java1.11
components:
  - referenceContent: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: ws
      spec:
        initContainers:
          - name: init-myservice
            image: adoptopenjdk/openjdk15-openj9
            command: ['sh', '-c', "echo copying jdk... && cp -R /opt/java/openjdk/ /tmp/jdk && echo done"]
    type: kubernetes
    volumes:
      - name: jdk
        containerPath: /tmp/jdk
        ephemeral: true
  - id: redhat/java11/latest
    preferences:
      java.configuration.runtimes: |-
        [
          {
            "name": "Open J9 15",
            "path": "/tmp/jdk/openjdk",
            "default": true
          }
        ]
    type: chePlugin
    volumes:
      - name: jdk
        containerPath: /tmp/jdk
  - mountSources: true
    memoryLimit: 512Mi
    type: dockerimage
    volumes:
      - name: jdk
        containerPath: /tmp/jdk
        ephemeral: true
      - name: m2
        containerPath: /home/user/.m2
    image: 'quay.io/eclipse/che-java11-maven:7.13.1'
    alias: maven
    env:
      - value: /tmp/jdk/openjdk
        name: JAVA_HOME
      - value: ''
        name: MAVEN_CONFIG
      - value: '-XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom'
        name: JAVA_OPTS
      - value: '-XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom -Duser.home=/home/user'
        name: MAVEN_OPTS
      - value: '-XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom'
        name: JAVA_TOOL_OPTIONS
commands:
  - name: maven build
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/console-java-simple'
        type: exec
        command: mvn clean install
        component: maven
  - name: maven build and run
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/console-java-simple'
        type: exec
        command: mvn clean install && java -jar ./target/*.jar
        component: maven

[mshaposhnik]

So does the volume sharing between init container and che-plugin container working?

[tsmaeder]

When I add a volume to the java meta.yaml like so:

      - mountPath: "/tmp/jdk"
        name: jdk
        ephemeral: true

the init container has load of errors like this one:

cp: cannot create regular file '/tmp/jdk/openjdk/legal/java.smartcardio/pcsclite.md': Permission denied

[ericwill]

@mshaposhnik please note that supporting ephemeral containers via the keyword ephemeral is important as well.

[mshaposhnik]

I have tested it using persistVolumes: false property in devfile (that means that all workspace volumes are ephemeral). Will try to elaborate permission error mentioned above.

[mshaposhnik]

@ericwill in fact what you requesting is completely another issue. You asking for ability so set ephemeral per-volume in devfile, not alltogether. Am i right ?

[ericwill]

@ericwill in fact what you requesting is completely another issue. You asking for ability so set ephemeral per-volume in devfile, not alltogether. Am i right ?

Sorry, perhaps it wasn't clear. Yes this is a requirement as well. To summarize, we need to:

• specify an ephemeral volume in the devfile which is visible to the che-plugin container,
• be able to copy from this volume, and
• be able to specify on a per-volume basis which volumes are ephemeral and which ones aren't.

[nickboldt]

7.21 is done, slip to 7.22

[mshaposhnik]

So closing that in favor of #18271 @skabashnyuk @ericwill ?

[ericwill]

So closing that in favor of #18271 @skabashnyuk @ericwill ?

No objections!

I do have a question though: when persistVolumes: false is specified, does the copying work? Or is it still broken?

[mshaposhnik]

@ericwill should work (i just have no plugins with ephemeral volumes to test)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.