Kubernetes + Persistent storage(Azure file) vs. Postgres = Fail!

[msj33]

Have tried now for 2 days to get a running postgres database up and running on Kubernetes with Azure files mountes as persistent volume - and it just does not work!

It works WITHOUT persistent storage - but as soon as I mount Azure files the pod end in a crashloop.

Have tried both Alpine and Debian in multiple versions:

I have tried the following on storageclass:

• dir_mode=0777
• file_mode=0777
• uid=999
• uid=999

in multiple combinations with setting this on the container
securityContext:
runAsUser: 999
fsGroup: 999

Even tried to chowning manually (without effec) AND trying with initContainers - but it just won't run

Note: The volume works(incl. setting ownership) if I mount it anywhere else than /var/lib/postgresql/data

Error from the logs:

1. Not setting anything or trying with root as owner/context
   initdb: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
2. • Mounting volume with postgres(999)
     2019-01-25 12:29:05.963 UTC [36] LOG: could not link file "pg_wal/xlogtemp.36" to "pg_wal/000000010000000000000001": Operation not supported
     2019-01-25 12:29:05.998 UTC [36] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory
     child process exited with exit code 1

I GIVE UP! 👎

The combination of these 3 components is far from simple or intuitive - Just reading the caveats on the README.md

Any good intentions of securing the database by an entryscript like this, just makes it unusable......but that also guarantees 100% security!?

Hope someone can help and/or make this installation working and simple - Else i'm about to spin up some BAREMETAL from the 90s to host a MSSQL database! :-)

_Sorry for the yelling..........just feeling stupid spending 2 full work days on this shit.....counterfeits the whole purpose of using docker _

[msj33]

My setup:

  containers:
  - name: my-db
    image: postgres:10.6-alpine
    env:
    - name: POSTGRES_USER
      value: "postgres"
    - name: POSTGRES_PASSWORD
      value: "testpassword"
    - name: PGDATA
      value: "/var/lib/postgresql/data/"
    - name: POSTGRES_DB
      value: "mydb"
    ports:
    - containerPort: 5432
    volumeMounts:
    - name: db-volume
      mountPath: "/var/lib/postgresql/data"
  volumes:
    - name: db-volume
      persistentVolumeClaim:
        claimName: my-db-data

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-db-data
spec:
accessModes:
- ReadWriteMany
storageClassName: azurefile-db
resources:
requests:
storage: 10Gi

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefile-db
provisioner: kubernetes.io/azure-file
mountOptions:

• dir_mode=0777
• file_mode=0777
• uid=999
• gid=999
  parameters:
  skuName: Standard_LRS
  storageAccount: dynamicfile
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
  name: my-db-data
  spec:
  accessModes:
  • ReadWriteMany
    storageClassName: azurefile-db
    resources:
    requests:
    storage: 10Gi

[wglambert]

I don't know if users to these repositories would be very familiar with Kubernetes, have you tried a Kubernetes forum or Stack Overflow? Since I don't think there's anything that could be changed in the image to alleviate this

[wglambert]

I did find kubernetes/minikube#2 which has your exact error: kubernetes/minikube#2 (comment)
Which seems to be related to Minikube being a VM and using shared folders, however I don't know its relevance to Azure

[yosifkit]

PostgreSQL does not work with Azure file. This is because PostgreSQL requires hard links in the Azure File directory, and since Azure File does not support hard links the pod fails to start.

https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/persistent_storage_azure_file.html

[msj33]

Damn........all this hard work for nothing - Could be good to know, that Azure files is not supported - Or that Postgres is dependant on hardlinks.

Thanks for a quick reponse though :-)

Will now have to look at other deployment options - Azure Disks(yak!) or maybe MariaDB for container solution.

[viniciusdanone]

I have tried to run MySQL and Postgre using StorageAccount, and I'm getted the same error. (permission for change directory). Your have tried again?

[flenoir]

Having same issue using MariaDB, does anyone have a solution ? a working database ?

[brunoscota]

I have tried to run MySQL and Postgre using StorageAccount, and I'm getted the same error. (permission for change directory). Your have tried again?

The problem only happen to postgres for me. I've tested with Mysql and it works fine.

using this mount point options for mysql

mountOptions:

• dir_mode=0777
• file_mode=0777
• uid=999
• gid=999
• mfsymlinks
• nobrl

[christianskovholm]

The issue occurs because Azure Files does not support hard links by default, which is required by postgres. You will need to create a NFS backed Azure File storage for it to work: https://docs.microsoft.com/en-us/azure/aks/azure-files-csi#nfs-file-shares

[asher-lab]

Thanks @christianskovholm your solution works!

[JelmerOffenberg]

Amazing @christianskovholm, this was really hard to find. This solved my issue, cheers.

[leepereira]

can someone please provide details on how this was resolved ?
Did you create the storage class for NFS ?
I am still seeing errors in the logs

`The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2023-01-31 00:56:48.933 UTC [38] LOG: could not link file "pg_wal/xlogtemp.38" to "pg_wal/000000010000000000000001": Operation not supported
2023-01-31 00:56:48.962 UTC [38] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ... % `

Can you help me with the manifest files for the pv and pvc that are used to create the share that is eventually mounted for the /var/lib/postgresql/data to be mounted from azure files