Two additional login errors on hot standby instance, followup to #3118

[kartiksubbarao]

Environment

• Python version: 3.5.2
• NetBox version: 2.5.12

Steps to Reproduce

1. Set up netbox on a remote system (master), and create a user account on that system
2. Set up postgresql on localhost as a hot standby replica of the master system database
3. Set up netbox on localhost. In configuration.py, set MAINTENANCE_MODE to True and SESSION_FILE_PATH to a writable directory.
4. Reload apache (or whatever WSGI server you have configured)
5. Try to login to netbox

Expected Behavior

I expected that logins would work successfully.

Observed Behavior

(This is copied from comments that I added to #3118 after it was closed. Perhaps they weren't seen?)

Logins fail. There seems to be at least two other places in the code, apart from what was addressed in #3118, where django tries to save information associated with a user login to the database. Here are the stack traces of those two cases:

Case 1

Authenticating with a local DB password. The key line seems to be line 110 of /usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py. Reading the comments, it seems to be associated with a password hash upgrade. But I also tried logging into the master instance (to flush out any password hash changes) and then logging in again to the standby instance. But I still get the same error.

Internal Server Error: /login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.InternalError: cannot execute UPDATE in a read-only transaction


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 32, in dispatch
    return super().dispatch(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 43, in post
    if form.is_valid():
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 185, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 180, in errors
    self.full_clean()
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 382, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 409, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/forms.py", line 196, in clean
    self.user_cache = authenticate(self.request, username=username, password=password)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/__init__.py", line 73, in authenticate
    user = backend.authenticate(request, **credentials)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/backends.py", line 26, in authenticate
    if user.check_password(password) and self.user_can_authenticate(user):
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py", line 111, in check_password
    return check_password(raw_password, self.password, setter)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/hashers.py", line 61, in check_password
    setter(password)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py", line 110, in setter
    self.save(update_fields=["password"])
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py", line 66, in save
    super().save(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1429, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.InternalError: cannot execute UPDATE in a read-only transaction

Case 2

Authenticating with an LDAP password. The key line here seems to be line 612 of /usr/local/lib/python3.5/dist-packages/django_auth_ldap/backend.py. It seems to think it needs to 'build' this user instead of just 'populating' it. Not sure what's going on here because I've already logged in as this user on the master instance several times. So whatever state it needs should have already been built in the database by now. Not sure why it wants to try to build state now on the standby instance.

Internal Server Error: /login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.InternalError: cannot execute UPDATE in a read-only transaction


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/utils/decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 32, in dispatch
    return super().dispatch(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/netbox/netbox/users/views.py", line 43, in post
    if form.is_valid():
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 185, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 180, in errors
    self.full_clean()
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 382, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.5/dist-packages/django/forms/forms.py", line 409, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/forms.py", line 196, in clean
    self.user_cache = authenticate(self.request, username=username, password=password)
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/__init__.py", line 73, in authenticate
    user = backend.authenticate(request, **credentials)
  File "/usr/local/lib/python3.5/dist-packages/django_auth_ldap/backend.py", line 150, in authenticate
    user = self.authenticate_ldap_user(ldap_user, password)
  File "/usr/local/lib/python3.5/dist-packages/django_auth_ldap/backend.py", line 210, in authenticate_ldap_user
    return ldap_user.authenticate(password)
  File "/usr/local/lib/python3.5/dist-packages/django_auth_ldap/backend.py", line 350, in authenticate
    self._get_or_create_user()
  File "/usr/local/lib/python3.5/dist-packages/django_auth_ldap/backend.py", line 612, in _get_or_create_user
    self._user.save()
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/base_user.py", line 66, in save
    super().save(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1429, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.5/dist-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.InternalError: cannot execute UPDATE in a read-only transaction

[jeremystretch]

Reading the comments, it seems to be associated with a password hash upgrade.

Agreed, this seems to be the root cause. Unfortunately there's nothing we can do to work around this reasonably.

But I also tried logging into the master instance (to flush out any password hash changes) and then logging in again to the standby instance. But I still get the same error.

I can confirm that this worked for me: After upgrading Django and logging into the master instance, I was able to log into the standby without issue. (I did also then confirm that the standby database is still in read-only mode; trying to create a new object raises an exception.)

You might want to double-check that the standby database is being updated by the master as expected. That is, check that the user account has the same exact hash value in both databases.

[kartiksubbarao]

I did a quick check of the password field in the auth_user table on both the master and standby and they're identical. I'll try to do some more troubleshooting when I get a chance.

Any ideas on the LDAP-related error (Case 2)?

[kartiksubbarao]

I did some further investigation and was able to resolve both issues:

Case 1 -- I logged in again to the master instance and logged in to the standby instance, and this time it worked. It successfully upgraded the hash on the master instance (increased the number of PBKDF2 iterations) and replicated the password to the standby instance. Not exactly sure why it didn't work previously, but it seems solid now.

Case 2 -- I was able to get this to work by setting AUTH_LDAP_ALWAYS_UPDATE_USER = False in ldap_config.py on the standby instance.