Worker will not re-connect anymore due to Celery 6.0 and above issue

[samip5]

Describe the bug
Worker might not re-connect if it fails enough of times anymore.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Authentik
2. Yeet Redis deployment for a moment
3. See error
4. De-deploy Redis
5. See issue with Worker not re-connecting

Expected behavior
I would expect it to re-connect.

Logs

{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning: \nIn Celery 5.1 we introduced an optional breaking change which\non connection loss cancels all currently executed tasks with late acknowledgement enabled.\nThese tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered\nback to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss\nsetting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.\n\n  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070849.2876282}
{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n  warnings.warn(\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070849.4042337}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 111 connecting to redis.databases.svc.cluster.local:6379. Connection refused..\nTrying again in 2.00 seconds... (1/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070849.4176264}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 1 connecting to redis.databases.svc.cluster.local:6379. Operation not permitted..\nTrying again in 4.00 seconds... (2/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070851.7179375}
{"event": "consumer: Cannot connect to redis://redis.databases.svc.cluster.local:6379/0: Error 1 connecting to redis.databases.svc.cluster.local:6379. Operation not permitted..\nTrying again in 6.00 seconds... (3/100)\n", "level": "error", "logger": "celery.worker.consumer.consumer", "timestamp": 1699070855.7518742}
{"event": "/ak-root/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine\nwhether broker connection retries are made during startup in Celery 6.0 and above.\nIf you wish to retain the existing behavior for retrying connections on startup,\nyou should set broker_connection_retry_on_startup to True.\n  warnings.warn(\n", "level": "warning", "logger": "py.warnings", "timestamp": 1699070862.060859}

Version and Deployment (please complete the following information):

• authentik version: 2023.10.2
• Deployment: Helm

[authentik-automation]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.