Postgres AKS persistent volume mapping problem

[n4ch04]

Using postgres in a AKS (Azure k8s Service) it is not possible to map a folder of the pod to a persistent volume. The contents of the mapped folder does not appear in pv.
If you manually create a file via azure portal into the pv you can't see it on the pod files and vice versa.
To reproduce it at first instance in aks cluster below is my setup:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-volume
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: azurefile
  resources:
    requests:
      storage: 1Gi

apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: database
        image: postgres:latest
        ports:
        - containerPort: 5432
        env:
            - name: POSTGRES_USER
              value: XXXXXXX
            - name: POSTGRES_PASSWORD
              value: XXXXXXX
            - name: POSTGRES_DB
              value: XXXXXX
            - name: PGDATA
              value: /var/lib/postgresql/data
 
        volumeMounts:
        - name: database
          mountPath: /var/lib/postgresql
          subPath: backup
      volumes:
      - name: database
        persistentVolumeClaim:
          claimName: database-volume

As you can see I create a PVC (using storageClass azurefile) used by a deployment to map a persistent volume to store the contents of the database (avoiding permissions problems setting subPath in the volumeMount).

I have tried to create the pv using manual config (not using pvc’s, creating the volume and accesing it with a secret into the deployment), using azureDisk instead of azureFile storage and the problem persists, the contents of the data folder are not mapped and if the pod dies, data dies with it.
I also tried to change default postgres data folder through PGDATA env variable, play with permissions (777) of data folder files, and issue is always there, looks like the volume is not correctly mapped.
Talk with azure support guys and they do not have registered this as azure storage issue, looks like only appears when using postgres image.

[wglambert]

#548 (comment)

PostgreSQL does not work with Azure file. This is because PostgreSQL requires hard links in the Azure File directory, and since Azure File does not support hard links the pod fails to start.

From https://docs.microsoft.com/en-us/azure/storage/files/storage-files-compare-protocols you might try the "NFS preview" which supports hard links, instead of SMB

[n4ch04]

I tried it, but appears mounting problems ... and it's has its cost.
People from azure tell me that this is a postgres problem, but i think it's a serious issue that you can't use postgres in aks ...

[n4ch04]

Any ideas are welcome, but I've tried every alternative I've found and it does not work

[n4ch04]

Oook, I think I have the solution.
Using azureDisk instead of azureFile and configuring a custom storageClass to use standard hdd (azure tries always to cheat you to use premium options) as shared volume and pointing pgdata at /var/lib/postgresql/backup finally works (maybe using ssd works too).
When I kill the pod files are still there when the deployment creates another replica
Below is my setup:

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-custom-storageclass
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  storageaccounttype: Standard_LRS
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "azuredisk-custom-storageclass"
  resources:
    requests:
      storage: 1Gi
---
kind: Deployment
metadata:
  name: database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: database
        image: postgres:latest
        ports:
        - containerPort: 5432
        env:
            - name: POSTGRES_USER
              value: 
            - name: POSTGRES_PASSWORD
              value: 
            - name: POSTGRES_DB
              value: 
            - name: PGDATA
              value: /var/lib/postgresql/backup
 
        volumeMounts:
          - name: database
            mountPath: /var/lib/postgresql
            subPath: backup
      
      volumes:
        - name: database
          persistentVolumeClaim:
            claimName: database-pvc

What is weird is despite db files are no longer stored in /var/lib/postgresql/data the pod have a mount on that path:

But there is not a filesystem mounted (at least as df shows):

I don't have time to research on this, but it looks like that partition could cause all of this problems ...
What is certain is that storing files on these partition is not a good idea.

[wglambert]

The /var/lib/postgresql/data is a VOLUME in the Dockerfile

postgres/13/Dockerfile

Line 183 in 14f13e4

VOLUME /var/lib/postgresql/data

There's a discussion around that in #404