_gdbm.error: Malformed database file header

[florian-rieder]

Describe the bug
Hi, I'm running the latest mhre/flower image in a k8s cluster (replicas=1). I'm running it in persistent mode, using a persistent volume claim for the database file. Flower seemed to work fine until this morning, when I noticed the web page returning a 503. The database file header seems to be malformed, and the container is now stuck in a crash loop.

# Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-flower
  template:
    metadata:
      labels:
        app: app-flower
    spec:
      securityContext:
        fsGroup: 1000
      containers:
      - name: app-flower
        image: myrepo/mher/flower
        imagePullPolicy: Always
        env:
        - name: CELERY_BROKER_URL
          value: redis://redis:6379/0
        - name: TZ
          value: Europe/Zurich
        - name: FLOWER_DB
          value: /usr/local/flowerdb/flower.db
        - name: FLOWER_PERSISTENT
          value: "true"
        - name: FLOWER_STATE_SAVE_INTERVAL
          value: '30000' # ms
        command: ["celery", "flower"]
        args:
          - "--url-prefix=flower"

        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
          capabilities:
            drop:
              - ALL
          runAsUser: 1000
          runAsGroup: 1000

        volumeMounts:
        - name: flower-pvc
          mountPath: /usr/local/flowerdb

      volumes:
      - name: flower-pvc
        persistentVolumeClaim:
          claimName: flower-pvc

# Persistent Volume Claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: flower-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeMode: Filesystem

Here's the full traceback:

Traceback (most recent call last):
File "/usr/local/bin/celery", line 8, in <module>
  sys.exit(main())
           ^^^^^^
File "/usr/local/lib/python3.11/site-packages/celery/__main__.py", line 15, in main
  sys.exit(_main())
             ^^^^^^^
File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 235, in main
  return celery(auto_envvar_prefix="CELERY")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
  return self.main(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
  rv = self.invoke(ctx)
       ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
  return ctx.invoke(self.callback, **ctx.params)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
  return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flower/command.py", line 48, in flower
  flower_app = Flower(capp=app, options=options, **settings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flower/app.py", line 57, in __init__
  self.events = events or Events(
                          ^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flower/events.py", line 134, in __init__
  state = shelve.open(self.db)
          ^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/shelve.py", line 243, in open
  return DbfilenameShelf(filename, flag, protocol, writeback)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/shelve.py", line 227, in __init__
  Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
                       ^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dbm/__init__.py", line 95, in open
  return mod.open(file, flag, mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
_gdbm.error: Malformed database file header

Expected behavior
The database file header shouldn't get corrupted. I think I can simply delete the database and start fresh but this is definitely less than ideal.

Is there an issue with my configuration ?