mkdir: can't create directory '/opt/redis': Permission denied

[enidvx]

Hello,
Following step by step installation and configuration here: https://ot-container-kit.github.io/redis-operator/#/installation/kubectl

I get the errors below when starting pods (on all of them)

mkdir: can't create directory '/opt/redis': Permission denied
/usr/bin/entrypoint.sh: line 14: /etc/redis/redis.conf: Permission denied
/usr/bin/entrypoint.sh: line 25: /etc/redis/redis.conf: Permission denied
sed: /data/nodes.conf: No such file or directory
/usr/bin/entrypoint.sh: line 45: /etc/redis/redis.conf: Permission denied
Starting redis service.....
11:C 07 Aug 2020 14:57:11.071 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:C 07 Aug 2020 14:57:11.071 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=11, just started
11:C 07 Aug 2020 14:57:11.071 # Configuration loaded
11:M 07 Aug 2020 14:57:11.073 * Running mode=standalone, port=6379.
11:M 07 Aug 2020 14:57:11.073 # Server initialized
11:M 07 Aug 2020 14:57:11.073 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
11:M 07 Aug 2020 14:57:11.165 * Ready to accept connections

Cluster is OKD4.5, and here are the cluster versions:

$ oc version
Client Version: 4.5.0-0.okd-2020-07-14-153706-ga
Server Version: 4.5.0-0.okd-2020-07-29-070316
Kubernetes Version: v1.18.3

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2-0-g52c56ce", GitCommit:"592b165", GitTreeState:"clean", BuildDate:"2020-07-14T05:27:34Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3", GitCommit:"012b3ec", GitTreeState:"clean", BuildDate:"2020-07-22T21:49:20Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

$ oc get pods
NAME                              READY   STATUS    RESTARTS   AGE
redis-master-0                    2/2     Running   0          39m
redis-master-1                    2/2     Running   0          39m
redis-master-2                    2/2     Running   0          39m
redis-operator-7885ff4477-g9mst   1/1     Running   0          41m
redis-slave-0                     2/2     Running   0          39m
redis-slave-1                     2/2     Running   0          39m
redis-slave-2                     2/2     Running   0          39m

$ oc get deployment
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
redis-operator   1/1     1            1           41m

[enidvx]

And also all PVs and PVCs are configured and on Bound status:

$ oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                                                 STORAGECLASS   REASON   AGE
redis-opstree-pv1   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-slave-redis-slave-0                          slow                    2d18h
redis-opstree-pv2   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-slave-redis-slave-2                          slow                    2d18h
redis-opstree-pv3   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-master-redis-master-0                        slow                    2d18h
redis-opstree-pv4   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-master-redis-master-2                        slow                    2d18h
redis-opstree-pv5   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-master-redis-master-1                        slow                    2d18h
redis-opstree-pv6   1Gi        RWO            Delete           Bound      redis-proj-opstree/redis-slave-redis-slave-1                          slow                    2d18h

$ oc get pvc
NAME                          STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-master-redis-master-0   Bound    redis-opstree-pv3   1Gi        RWO            slow           2d17h
redis-master-redis-master-1   Bound    redis-opstree-pv5   1Gi        RWO            slow           2d17h
redis-master-redis-master-2   Bound    redis-opstree-pv4   1Gi        RWO            slow           2d17h
redis-slave-redis-slave-0     Bound    redis-opstree-pv1   1Gi        RWO            slow           2d17h
redis-slave-redis-slave-1     Bound    redis-opstree-pv6   1Gi        RWO            slow           2d17h
redis-slave-redis-slave-2     Bound    redis-opstree-pv2   1Gi        RWO            slow           2d17h

Also on the persistent volume mounted on /data I can write from inside of any of the pods:

$ oc rsh redis-master-0
Defaulting container name to redis-master.
Use 'oc describe pod/redis-master-0 -n redis-proj-opstree' to see all of the containers in this pod.
/data $ 
/data $ pwd
/data
/data $ id
uid=1000610000(1000610000) gid=0(root) groups=1000610000
/data $ touch file.test
/data $ ls -lra
total 0
-rw-r--r--    1 10006100 root             0 Aug 10 08:07 file.test
drwxr-xr-x    1 root     root            28 Aug  7 14:57 ..
drwxrwxrwx    2 nobody   nobody          48 Aug 10 08:07 .

$ df -h /data
Filesystem                Size      Used Available Use% Mounted on
172.30.160.100:/var/nfsshare/redis_opstree
                         96.9G     35.4G     61.5G  36% /data

[enidvx]

And also these are the permissions on the directories above on all pods:

/data $ id
uid=1000610000(1000610000) gid=0(root) groups=1000610000
/data $ ls -ld /opt
drwxr-xr-x    2 root     root             6 Apr 23 13:10 /opt
/data $ ls -ld /etc/redis/
drwxr-xr-x    1 root     root            24 Jul 25 14:36 /etc/redis/
/data $ ls -l /etc/redis/redis.conf 
-rwxr-xr-x    1 root     root           130 Jul 25 14:31 /etc/redis/redis.conf

[enidvx]

The only way how I could make it work, without having any access or making any change on the images/pods directory permissions was by adding the scc policy like below which would cause PODs to run on elevated privileges as root. And this is not a good idea imo.
oc adm policy add-scc-to-user anyuid -z default

I wonder if there is any other way to make it run as default userid: 1000610000 without having to resort to this privilege escalation...

[iamabhishek-dubey]

Hmm, Ideally this shouldn't be coming, have you used any securityContext in parallel?

[enidvx]

Hmm, Ideally this shouldn't be coming, have you used any securityContext in parallel?

In my knowledge I haven't used any other securityContext or other configuration rather than the default ones. Could I check if any config is used other than the default?

[UlrichSchlueter]

Not sure this helps. Following this https://developers.redhat.com/blog/2020/10/26/adapting-docker-and-kubernetes-containers-to-run-on-red-hat-openshift-container-platform/#more-760987 I was able to get it work on crc locally:
Created my own image (in my local registry)
FROM opstree/redis:v2.0

RUN chgrp -R 0 /etc/redis &&
chmod -R g=u /etc/redis
RUN chgrp -R 0 /opt &&
chmod -R g=u /opt

and used that instead. Cluster started properly after that.

[Ruudieboy]

Facing the same issue on Openshift 4.5.18

[iamabhishek-dubey]

I guess this issue persists for openshift cluster, we will work on this

[iamabhishek-dubey]

Fixed in #26

[fkesgin]

Hi,
This issue seems still happening on OpenShift 4.6. Log file has following permission denied errors.

/usr/bin/entrypoint.sh: line 14: /etc/redis/redis.conf: Permission denied
/usr/bin/entrypoint.sh: line 25: /etc/redis/redis.conf: Permission denied

OpenShift Version 4.6.13
Kubernetes Version v1.19.0+3b01205
redis-operator.v0.4.0

[robermar23]

I can also confirm this is still an issue in OpenShift 4.6.

OpenShift Version 4.6.15
Kubernetes Version v1.19.0+1833054
redis-operator.v0.4.0

Can this be reopened?