New spring devfile does not work on default namespace

[maysunfaisal]

/kind bug

What versions of software are you using?

Operating System:

Output of odo version:

How did you run odo exactly?

odo push

Actual behavior

Maysuns-MacBook-Pro:springboot-ex maysun$ odo push

Validation
 ✓  Validating the devfile [57999ns]

Creating Kubernetes resources for component myjava
 ✓  Waiting for component to start [22s]

Applying URL changes
 ✓  URL myingress successfully deleted

Syncing to component myjava
 ✓  Checking file changes for pushing [4ms]
 ✓  Syncing files to the component [3s]

Executing devfile commands for component myjava
 ✗  Executing build command "mvn clean -Dmaven.repo.local=/mypath/.m2/repository package -Dmaven.test.skip=true" [2s]
 ✗  Failed to start component with name myjava. Error: Failed to create the component: command execution failed: unable to execute the run command: unable to exec command [/bin/sh -c mvn clean -Dmaven.repo.local=/mypath/.m2/repository package -Dmaven.test.skip=true]: 
[ERROR] Could not create local repository at /mypath/.m2/repository -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
: error while streaming command: command terminated with exit code 1

I manually tried it out inside the container, and looks like the container user does not have sufficient privilege to create dir:

I have no name!@myjava-556574d744-45hfg:/projects$ mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true
[ERROR] Could not create local repository at /home/user/.m2/repository -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
I have no name!@myjava-556574d744-45hfg:/projects$ cd /home/user/.m2/
I have no name!@myjava-556574d744-45hfg:~/.m2$ ls -la
total 20
drwxr-xr-x. 3 root root  4096 Sep 23 19:40 .
drwxrwxr-x. 1 root root    17 Sep 23 19:40 ..
drwx------. 2 root root 16384 Sep 23 19:40 lost+found
I have no name!@myjava-556574d744-45hfg:~/.m2$ mkdir test
mkdir: cannot create directory ‘test’: Permission denied
I have no name!@myjava-556574d744-45hfg:~/.m2$ whoami
whoami: cannot find name for user ID 10001
$ id
uid=10001 gid=0(root) groups=0(root)

Expected behavior

it should be able to execute mvn

Any logs, error output, etc?

[adisky]

@maysunfaisal please share the devfile you are using,
for registry one https://github.com/odo-devfiles/registry/blob/master/devfiles/java-springboot/devfile.yaml#L21 the directory gets created when we mount emptyDir volume, not when we execute build command.

[johnmcollier]

@maysunfaisal What storageclasses are being used on both clusters (kubectl get sc)? Likely permissions issues with the new non-root container and the storageclass that's being used for the m2 volume on the failing cluster.

Because Kubernetes mounts persistent-volumes as root-owned by default, some storage classes on non-OpenShift Kubernetes distributions have poor support for non-root containers.

[maysunfaisal]

@adisky its the default odo-devfile registry devfile.yaml. The repository dir gets created in .m2 when mvn executes and thats where it stores all the dependency downloads/cache.

@johnmcollier well this is the cluster-bot clusters. I'm not sure what the storage-class was yesterday but it looks like the devfile is working today! This is my storage-class now:

Maysuns-MacBook-Pro:springboot-ex maysun$ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   31m

I've noticed that when the new default devfile.yaml failed yesterday, the previous two container devfile using the hosted image from maysunfaisal passed. The difference was in the container. The che image did not have the write permissino for group, it was:

I have no name!@myjava-556574d744-45hfg:~/.m2$ ls -la
total 20
drwxr-xr-x. 3 root root  4096 Sep 23 19:40 .
drwxrwxr-x. 1 root root    17 Sep 23 19:40 ..
drwx------. 2 root root 16384 Sep 23 19:40 lost+found

Today, the same image is

I have no name!@myjava-9688b86fd-d4mjd:~/.m2$ ls -la
total 24
drwxrwsr-x.  4 root       1000570000  4096 Sep 24 14:50 .
drwxrwxr-x.  1 root       root          17 Sep 24 14:50 ..
drwxrws---.  2 root       1000570000 16384 Sep 24 14:49 lost+found
drwxr-sr-x. 19 1000570000 1000570000  4096 Sep 24 14:50 repository

[maysunfaisal]

So I've seen this again today and did a bit more research. It works fine for non default namespace and doesn't work well for default namespace.

Non default namespace:

Maysuns-MacBook-Pro:springboot-ex maysun$ odo push

Validation
 ✓  Validating the devfile [49230ns]

Creating Kubernetes resources for component myjava
 ✓  Waiting for component to start [27s]

Applying URL changes
 ✓  URL 8080-tcp: http://8080-tcp-myjava-mjf.apps.ci-ln-mqpgn9t-d5d6b.origin-ci-int-aws.dev.rhcloud.com/ created

Syncing to component myjava
 ✓  Checking files for pushing [5ms]
 ✓  Syncing files to the component [2s]

Executing devfile commands for component myjava
 ✓  Executing build command "mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true" [26s]
 ✓  Executing run command "mvn -Dmaven.repo.local=/home/user/.m2/repository spring-boot:run" [2s]

Pushing devfile component myjava
 ✓  Changes successfully pushed to component


Maysuns-MacBook-Pro:springboot-ex maysun$ oc exec -it myjava-b8d7cf49c-q7xsr bash
I have no name!@myjava-b8d7cf49c-q7xsr:/projects$ 
I have no name!@myjava-b8d7cf49c-q7xsr:/projects$ 
I have no name!@myjava-b8d7cf49c-q7xsr:/projects$ 
I have no name!@myjava-b8d7cf49c-q7xsr:/projects$ cd /home/user/.m2/
I have no name!@myjava-b8d7cf49c-q7xsr:~/.m2$ id
uid=1000570000 gid=0(root) groups=0(root),1000570000
I have no name!@myjava-b8d7cf49c-q7xsr:~/.m2$ ls -la
total 24
drwxrwsr-x.  4 root       1000570000  4096 Sep 28 17:09 .
drwxrwxr-x.  1 root       root          17 Sep 28 17:09 ..
drwxrws---.  2 root       1000570000 16384 Sep 28 17:09 lost+found
drwxr-sr-x. 19 1000570000 1000570000  4096 Sep 28 17:10 repository
I have no name!@myjava-b8d7cf49c-q7xsr:~/.m2$ mkdir test
I have no name!@myjava-b8d7cf49c-q7xsr:~/.m2$ ls -la
total 28
drwxrwsr-x.  5 root       1000570000  4096 Sep 28 17:11 .
drwxrwxr-x.  1 root       root          17 Sep 28 17:09 ..
drwxrws---.  2 root       1000570000 16384 Sep 28 17:09 lost+found
drwxr-sr-x. 19 1000570000 1000570000  4096 Sep 28 17:10 repository
drwxr-sr-x.  2 1000570000 1000570000  4096 Sep 28 17:11 test

Default namespace:

Maysuns-MacBook-Pro:springboot-ex maysun$ odo push

Validation
 ✓  Validating the devfile [59228ns]

Creating Kubernetes resources for component myjava
 ✓  Waiting for component to start [22s]

Applying URL changes
 ✓  URL 8080-tcp: http://8080-tcp-myjava-default.apps.ci-ln-mqpgn9t-d5d6b.origin-ci-int-aws.dev.rhcloud.com/ created

Syncing to component myjava
 ✓  Checking files for pushing [5ms]
 ✓  Syncing files to the component [1s]

Executing devfile commands for component myjava
 ✗  Executing build command "mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true" [2s]
 ✗  Failed to start component with name myjava. Error: Failed to create the component: command execution failed: unable to execute the run command: unable to exec command [/bin/sh -c mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true]: 
[ERROR] Could not create local repository at /home/user/.m2/repository -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
: error while streaming command: command terminated with exit code 1


Maysuns-MacBook-Pro:springboot-ex maysun$ oc get po
NAME                      READY     STATUS    RESTARTS   AGE
myjava-64f595d498-wh6g2   1/1       Running   0          35s
Maysuns-MacBook-Pro:springboot-ex maysun$ oc exec -it myjava-64f595d498-wh6g2 bash
I have no name!@myjava-64f595d498-wh6g2:/projects$ 
I have no name!@myjava-64f595d498-wh6g2:/projects$ 
I have no name!@myjava-64f595d498-wh6g2:/projects$ cd /home/user/.m2/
I have no name!@myjava-64f595d498-wh6g2:~/.m2$ ls -la
total 20
drwxr-xr-x. 3 root root  4096 Sep 28 17:13 .
drwxrwxr-x. 1 root root    17 Sep 28 17:13 ..
drwx------. 2 root root 16384 Sep 28 17:13 lost+found
I have no name!@myjava-64f595d498-wh6g2:~/.m2$ id
uid=10001 gid=0(root) groups=0(root)
I have no name!@myjava-64f595d498-wh6g2:~/.m2$ mkdir test
mkdir: cannot create directory ‘test’: Permission denied

So the official documentation from OCP is to give the root group sufficient permissions for arbitrary user ids - https://docs.openshift.com/container-platform/4.5/openshift_images/create-images.html#images-create-guide-openshift_create-images:

For an image to support running as an arbitrary user, directories and files that may be written to by processes in the image should be owned by the root group and be read/writable by that group. Files to be executed should also have group execute permissions.

Adding the following to your Dockerfile sets the directory and file permissions to allow users in the root group to access them in the built image:

RUN chgrp -R 0 /some/directory &&
chmod -R g=u /some/directory

Because the container user is always a member of the root group, the container user can read and write these files.

I was doing the same for my previous springboot image hosted on maysunfaisal, this was my Dockerfile for example:

FROM ibmjava:8-sfj

RUN mkdir -p /myproject

RUN chgrp -R 0 /myproject && \
    chmod -R g=u /myproject

COPY bin /artifacts/bin

Thx @johnmcollier for the doc link. This was documented in the google doc for devfile v1 stack authors, which I cant seem to find right now.

[maysunfaisal]

I would also like to point out that the pvc path is not owned by root group but the user group in non default namespace. Whereas in default namespace, everything is owned by root and does not have w for group

[maysunfaisal]

I've opened #4053 because this will be affecting any devfile.

[maysunfaisal]

This was discussed in the odo contributors meeting on Sept 29 and it was decided that we want odo to spit out a warning saying we're running in default namespace and things may not run as expected.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.