Kubernetes + Persistent storage(Azure file) vs. Postgres = Fail!

[msj33]

Have tried now for 2 days to get a running postgres database up and running on Kubernetes with Azure files mountes as persistent volume - and it just does not work!

It works WITHOUT persistent storage - but as soon as I mount Azure files the pod end in a crashloop.

Have tried both Alpine and Debian in multiple versions:

I have tried the following on storageclass:

• dir_mode=0777
• file_mode=0777
• uid=999
• uid=999

in multiple combinations with setting this on the container
securityContext:
runAsUser: 999
fsGroup: 999

Even tried to chowning manually (without effec) AND trying with initContainers - but it just won't run

Note: The volume works(incl. setting ownership) if I mount it anywhere else than /var/lib/postgresql/data

Error from the logs:

1. Not setting anything or trying with root as owner/context
   initdb: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
2. • Mounting volume with postgres(999)
     2019-01-25 12:29:05.963 UTC [36] LOG: could not link file "pg_wal/xlogtemp.36" to "pg_wal/000000010000000000000001": Operation not supported
     2019-01-25 12:29:05.998 UTC [36] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory
     child process exited with exit code 1

I GIVE UP! 👎

The combination of these 3 components is far from simple or intuitive - Just reading the caveats on the README.md

Any good intentions of securing the database by an entryscript like this, just makes it unusable......but that also guarantees 100% security!?

Hope someone can help and/or make this installation working and simple - Else i'm about to spin up some BAREMETAL from the 90s to host a MSSQL database! :-)

_Sorry for the yelling..........just feeling stupid spending 2 full work days on this shit.....counterfeits the whole purpose of using docker _

[msj33]

My setup:

  containers:
  - name: my-db
    image: postgres:10.6-alpine
    env:
    - name: POSTGRES_USER
      value: "postgres"
    - name: POSTGRES_PASSWORD
      value: "testpassword"
    - name: PGDATA
      value: "/var/lib/postgresql/data/"
    - name: POSTGRES_DB
      value: "mydb"
    ports:
    - containerPort: 5432
    volumeMounts:
    - name: db-volume
      mountPath: "/var/lib/postgresql/data"
  volumes:
    - name: db-volume
      persistentVolumeClaim:
        claimName: my-db-data

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-db-data
spec:
accessModes:
- ReadWriteMany
storageClassName: azurefile-db
resources:
requests:
storage: 10Gi

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefile-db
provisioner: kubernetes.io/azure-file
mountOptions:

• dir_mode=0777
• file_mode=0777
• uid=999
• gid=999
  parameters:
  skuName: Standard_LRS
  storageAccount: dynamicfile
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
  name: my-db-data
  spec:
  accessModes:
  • ReadWriteMany
    storageClassName: azurefile-db
    resources:
    requests:
    storage: 10Gi

[wglambert]

I don't know if users to these repositories would be very familiar with Kubernetes, have you tried a Kubernetes forum or Stack Overflow? Since I don't think there's anything that could be changed in the image to alleviate this

[wglambert]

I did find kubernetes/minikube#2 which has your exact error: kubernetes/minikube#2 (comment)
Which seems to be related to Minikube being a VM and using shared folders, however I don't know its relevance to Azure

[yosifkit]

PostgreSQL does not work with Azure file. This is because PostgreSQL requires hard links in the Azure File directory, and since Azure File does not support hard links the pod fails to start.

https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/persistent_storage_azure_file.html

[msj33]

Damn........all this hard work for nothing - Could be good to know, that Azure files is not supported - Or that Postgres is dependant on hardlinks.

Thanks for a quick reponse though :-)

Will now have to look at other deployment options - Azure Disks(yak!) or maybe MariaDB for container solution.

[viniciusdanone]

I have tried to run MySQL and Postgre using StorageAccount, and I'm getted the same error. (permission for change directory). Your have tried again?

[flenoir]

Having same issue using MariaDB, does anyone have a solution ? a working database ?

[brunoscota]

I have tried to run MySQL and Postgre using StorageAccount, and I'm getted the same error. (permission for change directory). Your have tried again?

The problem only happen to postgres for me. I've tested with Mysql and it works fine.

using this mount point options for mysql

mountOptions:

• dir_mode=0777
• file_mode=0777
• uid=999
• gid=999
• mfsymlinks
• nobrl

[christianskovholm]

The issue occurs because Azure Files does not support hard links by default, which is required by postgres. You will need to create a NFS backed Azure File storage for it to work: https://docs.microsoft.com/en-us/azure/aks/azure-files-csi#nfs-file-shares

[asher-lab]

Thanks @christianskovholm your solution works!

[JelmerOffenberg]

Amazing @christianskovholm, this was really hard to find. This solved my issue, cheers.

[leepereira]

can someone please provide details on how this was resolved ?
Did you create the storage class for NFS ?
I am still seeing errors in the logs

`The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2023-01-31 00:56:48.933 UTC [38] LOG: could not link file "pg_wal/xlogtemp.38" to "pg_wal/000000010000000000000001": Operation not supported
2023-01-31 00:56:48.962 UTC [38] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ... % `

Can you help me with the manifest files for the pv and pvc that are used to create the share that is eventually mounted for the /var/lib/postgresql/data to be mounted from azure files

[bansioza44]

Hi @christianskovholm I have tried https://docs.microsoft.com/en-us/azure/aks/azure-files-csi#nfs-file-shares as per your suggestion but in this doc there is no mention about storage account where fileshare available. Assuming It will create dynamic but it should create new fileshare right?
I am able to mount correctly with postgres datapath.

[elovin]

@leepereira You will have to disable "Secure transfer required" since it is not supported when using NFS to access the file shares of an azure Premium FileStorage account.

Go to Storage Account -> "Configuration" -> "Secure transfer required" -> Set to disabled

From the azure docs:

NFS file shares don't support an encryption mechanism, so in order to use the NFS protocol to access an Azure file share, you must disable require secure transfer for the storage account.

[koureasstavros]

The issue occurs because Azure Files does not support hard links by default, which is required by postgres. You will need to create a NFS backed Azure File storage for it to work: https://docs.microsoft.com/en-us/azure/aks/azure-files-csi#nfs-file-shares

I have to give many thanks to this post, I was struggling with this for more than 2-3 days.
In the end thinking about architecture and implementation, I am concluding those options:

• Deploying PostgreSQL in Kubernetes using Disks works and has multiple IOPs categories based on disks but there is a limitation on disks which can be attached on each Node.
• Deploying PostgreSQL in Kubernetes using Files works when (using NFS protocol) but has less IOPS options than disks but there is no limitation on the number of files which can be attached on each Node.

My final question, when using file.csi with protocol NFS we are using Azure Files or Azure NetApp files?

[mluds]

I fixed this with the following Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-csi-nfs-nonroot
provisioner: file.csi.azure.com
allowVolumeExpansion: true
# https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md
parameters:
  protocol: nfs
  networkEndpointType: privateEndpoint
reclaimPolicy: Delete
mountOptions:
  - nconnect=4
  - noresvport
  - actimeo=30

You also must give your AKS cluster Contributor access to the Resource Group. I used this Bicep code to do that (I have an AKS resource called aks):

// Grant AKS Contributor role on Resource Group for NFS Storage Class
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  scope: resourceGroup()
  name: guid(resourceGroup().id, aks.id)
  properties: {
    // Contributor
    // https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')
    principalId: aks.identity.principalId
    principalType: 'ServicePrincipal'
  }
}