permission denied errors in container logs for splunk

[aneesh786]

Hi Team,

Iam trying to install a cluster manager by using splunk operator 2.4.0. Containers are failing with below error.

tar: system/static/moreAppsIcon.png: Cannot open: No such file or directory
tar: system: Cannot mkdir: Permission denied
tar: system/static/splunkrc_cmds.xml: Cannot open: No such file or directory
tar: users: Cannot mkdir: Permission denied
tar: users: Cannot mkdir: Permission denied
tar: users/users.ini.default: Cannot open: No such file or directory
tar: Exiting with failure status due to previous errors

also looked at the container and looks like /opt/splunk/etc does n't have permission.
[splunk@splunk-cm-cluster-manager-0 splunk]$ ls -lrt
total 3220
-r--r--r--. 1 splunk splunk 85405 Aug 25 17:02 license-eula.txt
-r--r--r--. 1 splunk splunk 57 Aug 25 17:02 copyright.txt
-r--r--r--. 1 splunk splunk 520 Aug 25 17:05 README-splunk.txt
drwxr-xr-x. 3 splunk splunk 45 Aug 25 17:24 include
drwxr-xr-x. 2 splunk splunk 66 Aug 25 17:24 cmake
drwxr-xr-x. 2 splunk splunk 55 Aug 25 17:24 quarantined_files
drwxr-xr-x. 3 splunk splunk 58 Aug 25 17:24 openssl
-rw-r--r--. 1 splunk splunk 0 Aug 25 17:24 ftr
drwxr-xr-x. 2 splunk splunk 54 Aug 25 17:24 swidtag
-r--r--r--. 1 splunk splunk 3200489 Aug 25 17:25 splunk-9.1.1-64e843ea36b1-linux-2.6-x86_64-manifest
drwxr-xr-x. 1 splunk splunk 20 Sep 8 18:14 share
drwxr-xr-x. 1 splunk splunk 43 Sep 8 18:14 lib
drwxr-xr-x. 1 splunk splunk 169 Sep 8 18:14 bin
drwxrwxrwx. 2 root root 6 Nov 9 18:39 var
drwxrwxrwx. 2 root root 6 Nov 9 18:39 etc
[splunk@splunk-cm-cluster-manager-0 splunk]$

I tried standalone deployment and that too failed with same error. Do we have any fix for this ?

[vivekr-splunk]

Hi @aneesh786 can you share your cluster manager CR.

[aneesh786]

Hi @vivekr-splunk,

Thank you for replying to my issue.
While trying to create cluster manager or any resource, my container was unable to mount /etc and /var.
I had configured persistent storage using local-path. I believe this was the problem. Pod was unable to create volumes with this storage.
Created another storage class using ceph cluster and made it as default storage class. This resolved the issue.

[aneesh786]

Closing the discussion as issue is resolved