Background Tasks UI Broken when using Redis Sentinel

[tagur87]

Deployment Type

Self-hosted

NetBox Version

v3.7.0

Python Version

3.11

Steps to Reproduce

1. Deploy Redis Sentinel instance
2. Deploy netbox with the following configuration for the REDIS_KEY

REDIS = {
  'tasks': {
      'SENTINELS': [('<sentinel>', <port>)],
      'SENTINEL_SERVICE': 'mymaster',
      'SENTINEL_TIMEOUT': 10,
      'PASSWORD': '',
      'DATABASE': 0,
      'SSL': False,
      'INSECURE_SKIP_TLS_VERIFY': False,
  },
  'caching': {
      'SENTINELS': [('<sentinel>', <port>)],
      'SENTINEL_SERVICE': 'mymaster',
      'SENTINEL_TIMEOUT': 10,
      'PASSWORD': '',
      'DATABASE': 1,
      'SSL': False,
      'INSECURE_SKIP_TLS_VERIFY': False,
  },
}

3. Browse to http://..../admin/background-tasks/
4. See error message

Expected Behavior

Background tasks page should work as normal when a regular redis instance is used.

Observed Behavior

Following error is returned from debug output:

Environment:


Request Method: GET
Request URL: https://admin-jdt-nsot.o11n-dev.jw-cd-lab-1.10aws.org/admin/background-tasks/

Django Version: 4.2.8
Python Version: 3.11.4
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.forms',
 'corsheaders',
 'debug_toolbar',
 'graphiql_debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'graphene_django',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'core',
 'account',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'vpn',
 'wireless',
 'django_rq',
 'drf_spectacular',
 'drf_spectacular_sidecar',
 'netbox_ip_reservations.IPReservationConfig']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'netbox.middleware.RemoteUserMiddleware',
 'netbox.middleware.CoreMiddleware',
 'netbox.middleware.MaintenanceModeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapper_view_func
    response = view_func(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/views.py", line 37, in stats
    **get_scheduler_statistics(),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/utils.py", line 106, in get_scheduler_statistics
    conn_key = f"{connection_kwargs['host']}:{connection_kwargs['port']}/{connection_kwargs.get('db', 0)}"
                  ^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: KeyError at /admin/background-tasks/
Exception Value: 'host'

[moonrail]

We run into this issue as well - it seems to have been introduced in django-rq 2.9.0 via this commit:
rq/django-rq@df32713

Not completely sure, but seeing our own settings, your settings and the examples of django-rq in their README.md it seems they did not test with Sentinel and simply broke support with this change.

Edit: We've downgraded django-rq to 2.8.1 and there the error does not occur. If someone else might verify this as well, we should open an issue at django-rq´s repository.

[jeremystretch]

@tagur87 @moonrail could one of you please raise a bug report with django-rq to confirm that this is an uptsream issue?

[arthanson]

There is a ticket in django-rq: rq/django-rq#642

[moonrail]

For NetBox < 4.0.0 this issue is relevant, as the Django Admin UI for Background Tasks is affected.

Since NetBox 4.0.0 this Admin UI is not integrated anymore and a Custom UI was built directly into NetBox. This custom implementation does not seem to call the function django_rq.utils.get_scheduler_statistics which contains the bug.

Therefore NetBox >= 4.0.0 currently does not hit this django-rq bug.

The underlying bug at django-rq has been fixed via rq/django-rq@77bc4ff but is not released yet (current version is 2.10.2).

[jeremystretch]

NetBox v4.1.7 upgraded django-rq to v3.0, presumably resolving the upstream issue with Sentinel support. @moonrail @tagur87 are either of you able to confirm this?