Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RO file system #545

Closed
jimmi4u opened this issue Nov 6, 2022 · 30 comments
Closed

RO file system #545

jimmi4u opened this issue Nov 6, 2022 · 30 comments

Comments

@jimmi4u
Copy link

jimmi4u commented Nov 6, 2022

Hi,
I'm trying to run your chart via helm and am getting an error, no matter which chart version I choose:

INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']                                                                                 
executing as current user                                                                                                                                                                                                                              If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide                                                                                                                                                                                                                                                                                                                                           Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out                                                                                                                                                                        ---------------------------------------------------------------------------                                                                                                                                                                           Using Java: /opt/java/openjdk/bin/java                                                                                                                                                                                                                log4j:ERROR setFile(null,true) call failed.                                                                                                                                                                                                            java.io.FileNotFoundException: /opt/atlassian/confluence/logs/synchrony-proxy-watchdog.log (Read-only file system)                                                                                                                                         at java.base/java.io.FileOutputStream.open0(Native Method)                                                                                                                                                                                             at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)                                                                                                                                                                                 at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)                                                                                                                                                                               at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:158)                                                                                                                                                                                at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)                                                                                                                                                                                        at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)                                                                                                                                                                          at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)                                                                                                                                                                                at com.atlassian.confluence.bootstrap.SynchronyProxyWatchdog.addLogFileAppender(SynchronyProxyWatchdog.java:106)                                                                                                                                       at com.atlassian.confluence.bootstrap.SynchronyProxyWatchdog.main(SynchronyProxyWatchdog.java:47)                                                                                                                                                  2022-11-06 20:11:09,875 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required      ---------------------------------------------------------------------------                                                                                                                                                                            NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun. 
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-11-06_20-11-09.log instead.                                                                                                                     
 [0.001s][error  ][logging] Error opening log file '/opt/atlassian/confluence/logs/gc-2022-11-06_20-11-09.log': Read-only file system                                                                                                                   
[0.001s][error  ][logging] Initialization of output 'file=/opt/atlassian/confluence/logs/gc-2022-11-06_20-11-09.log' using options 'filecount=5,filesize=2M' failed.                                                                                 
 Invalid -Xlog option '-Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-1022-11-06_20-11-09.log::filecount=5,filesize=2M', see error log for details.                                                                                        Error: Could not create the Java Virtual Machine.                                                                                                                                                                                                      Error: A fatal exception has occurred. Program will exit.                                                                                                                                                                                              Stream closed EOF for confluence/confluence-confluence-confluence-server-5c6b654f78-9fkv7 (confluence) 

here are my values:

service:
      type: LoadBalancer
      port: 8090
    persistence:
      enabled: true
      storageClass: nfs-client
    securityContext: 
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1001
    psql:
      host: confluence-primary.confluence.svc
      port: 5432
      database: confluence
      username: confluence
      password:
        secret: confluence-pguser-confluence
        key: password

need some help please!
Thanks,
jim

@stevehipwell
Copy link
Owner

Hey @jimmi4u it looks like you're trying to make the pod more secure through a custom securityContext, which I completely approve of but I'm not sure if it's possible. There is a reason why the chart defaults are what they are and not more secure by default (lack of documentation or engagement from Atlassian).

If you just want to make the root filesystem read only you might be able to make use of the extra volumes & mounts to add emptyDir volumes where temporary files are being written (I suspect /tmp is used here).

If you want to make further security improvements I'd suggest that you have 2 paths to follow for this; path 1 would be to look at the image details and speak to Atlassian to see which security settings are compatible with their image, path 2 would be to use a security monitoring tool such as Falco to see what the container is doing which would conflict with your desired security settings.

If you choose to look further into this and have some findings which can be added to the Helm chart, such as adding an emptyDir for /tmp, please let me know and I'll make the changes to support them.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

Hi @stevehipwell, thanks for your reply! Btw. the security settings used are the defaults. I played a bit with them because of that error. So with or without them the error still occurs. What is also interesting is that the helmrelease reconcilation is still in progress so this happens right at the beginning and the pod is going CrashLoopBackOff....
I'll play a bit with the secure settings and the extra mounts. But if you have some other suggestions please let me know.
Thanks!

Edit: with default settings I get:

WARNING:root:Could not chown path /var/atlassian/application-data/confluence to confluence:confluence due to insufficient permissions.                                                                                                                
INFO:root:User is currently root. Will change directory ownership and downgrade run user to confluence                                                                                                                                                 INFO:root:User is currently root. Will downgrade run user to confluence                                                                                                                                                                               Traceback (most recent call last): 
File "/entrypoint.py", line 20, in <module>
exec_app([f'{CONFLUENCE_INSTALL_DIR}/bin/start-confluence.sh', '-fg'], CONFLUENCE_HOME,    
File "/entrypoint_helpers.py", line 146, in exec_app  write_pidfile()                                                                                                                                                                                                                                      File "/entrypoint_helpers.py", line 128, in write_pidfile with open(pidfile, 'wt', encoding='utf-8') as fd: 
PermissionError: [Errno 13] Permission denied: '/var/atlassian/application-data/confluence/docker-app.pid

@stevehipwell
Copy link
Owner

@jimmi4u are you sure you've not changed any of the security context settings? The chart is tested before releasing so it definitely runs, although your cluster may be configured differently to Kind (it should be) but I would expect the chart to work in most cases.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

@stevehipwell there is nothing else, except my PostgresCluster.
this is my helmrelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: confluence
  namespace: confluence
spec:
  chart:
    spec:
      chart: confluence-server
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: confluence
        namespace: confluence
  interval: 168h # = 1w
  targetNamespace: confluence
  values:
    service:
      type: LoadBalancer
      port: 8090
    persistence:
      enabled: true
      storageClass: nfs-client
    psql:
      host: confluence-primary.confluence.svc
      port: 5432
      database: confluence
      username: confluence
      password:
        secret: confluence-pguser-confluence
        key: password

@stevehipwell
Copy link
Owner

@jimmi4u could you confirm that you're not setting anything for securityContext? I think there are some GH formatting issues. The error you're getting looks to be related to setting the user to non-root so it can't chown and switch to the confluence user.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

@stevehipwell yep, there is no secrutiyContext!

@stevehipwell
Copy link
Owner

@jimmi4u what do you get as an output from helm get values? And could you share the Deployment config which results?

FYI if this is a new deployment I'd suggest setting deployment to false so you get a StatefulSet.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

@stevehipwell I'm using flux with helm so I don't have helm installed. Deployment is not ready, becuase of the pod error. Here is his description:
Name: confluence-confluence-confluence-server ││ Namespace: confluence ││ CreationTimestamp: Mon, 07 Nov 2022 17:51:17 +0100 ││ Labels: app.kubernetes.io/instance=confluence-confluence ││ app.kubernetes.io/managed-by=Helm ││ app.kubernetes.io/name=confluence-server ││ app.kubernetes.io/version=7.20.1 ││ helm.sh/chart=confluence-server-4.11.1 ││ helm.toolkit.fluxcd.io/name=confluence ││ helm.toolkit.fluxcd.io/namespace=confluence ││ Annotations: deployment.kubernetes.io/revision: 1 ││ meta.helm.sh/release-name: confluence-confluence ││ meta.helm.sh/release-namespace: confluence ││ Selector: app.kubernetes.io/instance=confluence-confluence,app.kubernetes.io/name=confluence-server ││ Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable ││ StrategyType: Recreate ││ MinReadySeconds: 0 ││ Pod Template: ││ Labels: app.kubernetes.io/instance=confluence-confluence ││ app.kubernetes.io/name=confluence-server ││ Service Account: confluence-confluence-confluence-server ││ Containers: ││ confluence: ││ Image: atlassian/confluence-server:7.20.1-jdk11 ││ Ports: 8090/TCP, 8091/TCP ││ Host Ports: 0/TCP, 0/TCP ││ Liveness: tcp-socket :http delay=60s timeout=1s period=30s #success=1 #failure=10 ││ Readiness: http-get http://:http/status delay=60s timeout=1s period=30s #success=1 #failure=10 ││ Environment: ││ JVM_MINIMUM_MEMORY: 512M ││ JVM_MAXIMUM_MEMORY: 512M ││ JVM_SUPPORT_RECOMMENDED_ARGS: -XX:+UseG1GC -Dsynchrony.memory.max=0m ││ ATL_JDBC_URL: jdbc:postgresql://confluence-primary.confluence.svc:5432/confluence ││ ATL_JDBC_USER: confluence ││ ATL_JDBC_PASSWORD: <set to the key 'password' in secret 'confluence-pguser-confluence'> Optional: false ││ ATL_DB_TYPE: postgresql ││ Mounts: ││ /var/atlassian/application-data/confluence from confluence-confluence-confluence-server-data (rw) ││ Volumes: ││ confluence-confluence-confluence-server-data: ││ Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ││ ClaimName: confluence-confluence-confluence-server-data ││ ReadOnly: false ││ Conditions: ││ Type Status Reason ││ ---- ------ ------ ││ Available False MinimumReplicasUnavailable ││ Progressing False ProgressDeadlineExceeded ││ OldReplicaSets: <none> ││ NewReplicaSet: confluence-confluence-confluence-server-5bb9686f49 (1/1 replicas created) ││ Events: ││ Type Reason Age From Message ││ ---- ------ ---- ---- ------- ││ Normal ScalingReplicaSet 26m deployment-controller Scaled up replica set confluence-confluence-confluence-server-5bb9686f49 to 1
btw. how can I make this more readable?

@stevehipwell
Copy link
Owner

@jimmi4u that's not enough to go on, surely you can run kubectl --namespace confluence get deployment confluence-confluence-confluence-server --output yaml? You'd also probably be better off debugging this directly rather than through Flux.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

yeah, you're right:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: confluence-confluence
    meta.helm.sh/release-namespace: confluence
  creationTimestamp: "2022-11-07T17:45:42Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: confluence-confluence
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: confluence-server
    app.kubernetes.io/version: 7.20.1
    helm.sh/chart: confluence-server-4.11.1
    helm.toolkit.fluxcd.io/name: confluence
    helm.toolkit.fluxcd.io/namespace: confluence
  name: confluence-confluence-confluence-server
  namespace: confluence
  resourceVersion: "2258574"
  uid: b21364e9-90a6-478b-ac2b-1a7adb8233ca
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: confluence-confluence
      app.kubernetes.io/name: confluence-server
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: confluence-confluence
        app.kubernetes.io/name: confluence-server
    spec:
      containers:
      - env:
        - name: JVM_MINIMUM_MEMORY
          value: 512M
        - name: JVM_MAXIMUM_MEMORY
          value: 512M
        - name: JVM_SUPPORT_RECOMMENDED_ARGS
          value: -XX:+UseG1GC  -Dsynchrony.memory.max=0m
        - name: ATL_JDBC_URL
          value: jdbc:postgresql://confluence-primary.confluence.svc:5432/confluence
        - name: ATL_JDBC_USER
          value: confluence
        - name: ATL_JDBC_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: confluence-pguser-confluence
        - name: ATL_DB_TYPE
          value: postgresql
        image: atlassian/confluence-server:7.20.1-jdk11
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 10
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        name: confluence
        ports:
        - containerPort: 8090
          name: http
          protocol: TCP
        - containerPort: 8091
          name: synchrony
          protocol: TCP
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /status
            port: http
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/atlassian/application-data/confluence
          name: confluence-confluence-confluence-server-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2002
      serviceAccount: confluence-confluence-confluence-server
      serviceAccountName: confluence-confluence-confluence-server
      terminationGracePeriodSeconds: 30
      volumes:
      - name: confluence-confluence-confluence-server-data
        persistentVolumeClaim:
          claimName: confluence-confluence-confluence-server-data
status:
  conditions:
  - lastTransitionTime: "2022-11-07T17:45:43Z"
    lastUpdateTime: "2022-11-07T17:45:43Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-11-07T17:45:42Z"
    lastUpdateTime: "2022-11-07T17:45:43Z"
    message: ReplicaSet "confluence-confluence-confluence-server-5bb9686f49" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

thanks!

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

and here again the error:

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2 
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2                             
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2                        
INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.                                                                                                                                                                                                               WARNING:root:Could not chown path /var/atlassian/application-data/confluence to confluence:confluence due to insufficient permissions.                     
WARNING:root:Could not chown path /var/atlassian/application-data/confluence to confluence:confluence due to insufficient permissions.        
WARNING:root:Could not chown path /var/atlassian/application-data/confluence to confluence:confluence due to insufficient permissions.                     
INFO:root:User is currently root. Will change directory ownership and downgrade run user to confluence
INFO:root:User is currently root. Will downgrade run user to confluence                                                                                                                                                                                                                                 Traceback (most recent call last):  
File "/entrypoint.py", line 20, in <module>  exec_app([f'{CONFLUENCE_INSTALL_DIR}/bin/start-confluence.sh', '-fg'], CONFLUENCE_HOME,
File "/entrypoint_helpers.py", line 146, in exec_app  write_pidfile()                                                                                                                                                                                                                                                                                     File "/entrypoint_helpers.py", line 128, in write_pidfile                                                                                                                                                                                                                                                 with open(pidfile, 'wt', encoding='utf-8') as fd:
PermissionError: [Errno 13] Permission denied: '/var/atlassian/application-data/confluence/docker-app.pid

@stevehipwell
Copy link
Owner

@jimmi4u last questions; what Kubernetes version, distro and addons for CRI, CNI & CSI?

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

k version --output=yaml

clientVersion:
  buildDate: "2022-09-28T16:52:07Z"
  compiler: gc
  gitCommit: a8e0c66d1a90a2bbc4ffa975129ca35756cc7c14
  gitTreeState: clean
  gitVersion: v1.24.6+k3s1
  goVersion: go1.18.6
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-09-28T16:52:07Z"
  compiler: gc
  gitCommit: a8e0c66d1a90a2bbc4ffa975129ca35756cc7c14
  gitTreeState: clean
  gitVersion: v1.24.6+k3s1
  goVersion: go1.18.6
  major: "1"
  minor: "24"
  platform: linux/amd64

uname -a:
Linux master 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

kube system with coredns, local-path-provisioner, metrics-server, traefik, Flannel, MetalLB 0.13, FluxCD2 and
nfs-subdir-external-provisioner, crunchydata-pgo, redis-op on 3 nodes. that's all.

@stevehipwell
Copy link
Owner

@jimmi4u I think it sounds like you've got the issue described in kubernetes-sigs/nfs-subdir-external-provisioner#173.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

@stevehipwell thanks for your help and for pointing me in the right direction. I realized, that I forgot to apply the pvc for confluence. Now I have it like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: confluence
  name: confluence-pvc
  namespace: confluence
  annotations:
    nfs.io/createUID: "1000" # set folder uid as createUID on creation, not required, default 0 (root)
    nfs.io/createGID: "1000" # set folder gid as createGID on creation, not required, default 0 (root)
    nfs.io/createMode: "0777" # set folder mode as createMode on creation, not required, default 0777 (a+rwx)
spec:
  storageClassName: vpig-managed
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

and the helmrelease like this:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: confluence
  namespace: confluence
spec:
  chart:
    spec:
      chart: confluence-server
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: confluence
        namespace: confluence
  interval: 168h
  targetNamespace: confluence
  values:
    serviceAccount: 
      create: true
    service:
      type: LoadBalancer
      port: 8090
    podSecurityContext:
      fsGroup: 2002 
    securityContext: 
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 1000
    persistence:
      enabled: true
      existingClaim: confluence-pvc
    psql:
      host: confluence-primary.confluence.svc
      port: 5432
      database: confluence
      username: confluence
      password:
        secret: confluence-pguser-confluence
        key: password

but am getting the same error like in the beginning:

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
WARNING:root:Permission problem writing '/opt/atlassian/confluence/conf/server.xml'; skipping
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
WARNING:root:Permission problem writing '/opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml'; skipping
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
WARNING:root:Permission problem writing '/opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties'; skipping
INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']
executing as current user
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/java/openjdk/bin/java
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /opt/atlassian/confluence/logs/synchrony-proxy-watchdog.log (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:158)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at com.atlassian.confluence.bootstrap.SynchronyProxyWatchdog.addLogFileAppender(SynchronyProxyWatchdog.java:106)
    at com.atlassian.confluence.bootstrap.SynchronyProxyWatchdog.main(SynchronyProxyWatchdog.java:47)
2022-11-07 20:49:13,586 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.002s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-11-07_20-49-13.log instead.
[0.002s][error  ][logging] Error opening log file '/opt/atlassian/confluence/logs/gc-2022-11-07_20-49-13.log': Permission denied
[0.002s][error  ][logging] Initialization of output 'file=/opt/atlassian/confluence/logs/gc-2022-11-07_20-49-13.log' using options 'filecount=5,filesize=2M' failed.
Invalid -Xlog option '-Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2022-11-07_20-49-13.log::filecount=5,filesize=2M', see error log for details.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Stream closed EOF for confluence/confluence-confluence-confluence-server-bc85b8dff-mrdsx (confluence)

when I unset the securityContext I'm getting the error as before...
And the funny thing is that all other apps are running fine with the storageClass, just not confluence...
I appreciate your help!

@jimmi4u
Copy link
Author

jimmi4u commented Nov 7, 2022

I looked into the values.yaml of the official atlassian datacenter and they have a:

nfsPermissionFixer:

  # -- If 'true', this will alter the shared-home volume's root directory so that Confluence
  # can write to it. This is a workaround for a K8s bug affecting NFS volumes:
  # https://github.com/kubernetes/examples/issues/260
  #
  enabled: true

I tried also other confluence repos on artifacthub with the same conclusion.
Have to try the official repo....

@stevehipwell
Copy link
Owner

@jimmi4u what does that flag do? If it's just a chown from an init container you can add that directly through the values.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

I also think it's a chown. How can I add it to the values. how can I also add an umask?
so how do values from the official repo translate to this repo values? f.e. volumes.sharedHome.nfsPermissionFixer.enabled?
thanks!

@stevehipwell
Copy link
Owner

This repo has nothing to do with the official Atlassian repo and predates it by a number of years (I did try to donate contribute my charts to them a number of times).

The issue you mentioned above (kubernetes/examples#260) has an example of the init container you'd need, this is a limitation of the CSI driver you're using so you'll need to figure out exactly what implementation you need. To configure it with this chart you'd add a container spec to extraInitContainers to do the work for you (example below).

extraInitContainers:
  - name: nfs-fix
    image: alpine:3.16.2
    securityContext:
      runAsUser: 0
    command: ["/bin/sh"]
    args: ["-c", "chgrp 2002 /var/atlassian/application-data/confluence; chmod g+w /var/atlassian/application-data/confluence"]
    volumeMounts:
      - mountPath: /var/atlassian/application-data/confluence
        name: confluence-server-data

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

@stevehipwell you're awesome! thanks so much!
I have troubles with the volumeMount. I tried to mount the pv from the pvc with its name, but helmrelease can't find it.
warning: Upgrade "confluence-confluence" failed: failed to create resource: Deployment.apps "confluence-confluence-confluence-server" is invalid: spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "pvc-b9adf5e7-6a28-4a9b-950b-fe1cfbd8c03e"
Do you have a hint, how to get access to the Volume?

@stevehipwell
Copy link
Owner

@jimmi4u it depends on the name you're using for the Helm deployment, looking above I think you'd need to use confluence-confluence-server-data; this isn't a PVC name/id it's the volume name.

FYI I assume you're setting the PVC you created externally in the chart values via persistence.existingClaim?

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

Hi @stevehipwell,
I already tried it with confluence-confluence-server-data but got the error so I tried with the pvc one....
How can I find out the correct volume's name?
You are right with the PVC setting...
Thanks so much for your time and affort!

@stevehipwell
Copy link
Owner

@jimmi4u just look at the deployment volumes and the correct name will be in there and in the main container volumeMounts.

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

hi, ok I got it!
but now with theis error:

confluence INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2
confluence INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
confluence INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2
confluence INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.
Stream closed EOF for confluence/confluence-confluence-confluence-server-74698c94cc-7d7gq (nfs-fix)
confluence INFO:root:User is currently root. Will change directory ownership and downgrade run user to confluence
confluence INFO:root:User is currently root. Will downgrade run user to confluence
confluence Traceback (most recent call last):
confluence   File "/entrypoint.py", line 20, in <module>
confluence     exec_app([f'{CONFLUENCE_INSTALL_DIR}/bin/start-confluence.sh', '-fg'], CONFLUENCE_HOME,
confluence   File "/entrypoint_helpers.py", line 146, in exec_app
confluence     write_pidfile()
confluence   File "/entrypoint_helpers.py", line 128, in write_pidfile
confluence     with open(pidfile, 'wt', encoding='utf-8') as fd:
confluence PermissionError: [Errno 13] Permission denied: '/var/atlassian/application-data/confluence/docker-app.pid'
Stream closed EOF for confluence/confluence-confluence-confluence-server-74698c94cc-7d7gq (confluence)

I really checked everything, the nfs server setting, rights, mods, recreated pvcs, added new StorageClass... no luck...
All other pods are running as expected...

and here is my helmrelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: confluence
  namespace: confluence
spec:
  chart:
    spec:
      chart: confluence-server
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: confluence
        namespace: confluence
  interval: 168h # = 1w
  targetNamespace: confluence
  values:
    serviceAccount: 
      create: true
    service:
      type: LoadBalancer
      port: 8090
   persistence:
      enabled: true
      storageClass: nfs-client
    psql:
      host: confluence-primary.confluence.svc
      port: 5432
      database: confluence
      username: confluence
      password:
        secret: confluence-pguser-confluence
        key: password
    extraInitContainers:
      - name: nfs-fix
        image: alpine:3.16.2
        securityContext:
          runAsUser: 0
        command: ["/bin/sh"]
        args: ["-c", "chgrp 2002 /var/atlassian/application-data/confluence; chmod g+w /var/atlassian/application-data/confluence"]
        volumeMounts:
          - mountPath: /var/atlassian/application-data/confluence
            name: confluence-confluence-confluence-server-data

@stevehipwell
Copy link
Owner

It looks like you're hitting this issue (there are other threads on Google which you might also want to explore).

https://community.atlassian.com/t5/Confluence-questions/Confluence-docker-container-changes-permission-of-own-directory/qaq-p/1149841

@stevehipwell
Copy link
Owner

Try setting SET_PERMISSIONS to false in the env value.

env:
  - name: SET_PERMISSIONS
    value: "true"

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

@stevehipwell that did the trick! You're amazing! Thanks a lot!
But now I have to realize that I'm in the need of a license key... no way for personal use ... thought it was the old CE... what a pitty....

@stevehipwell
Copy link
Owner

This is the community edition but I don't think you can get new licences anymore, paid or trial.

https://www.atlassian.com/migration/assess/journey-to-cloud

@jimmi4u
Copy link
Author

jimmi4u commented Nov 8, 2022

hmm, very sad...
But as mentionend before: thank you for your great support and your time! At least I have learned something ;)
Edit: and there is no way to get an old one?

@stevehipwell
Copy link
Owner

You're welcome @jimmi4u, would you mind closing this issue?

@jimmi4u jimmi4u closed this as completed Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants