Reduce the chance of database locks interrupting soud syncing.

[rtibbles]

Summary

Wraps the SyncQueue save method in the retry_on_db_lock decorator to reduce the chances of interrupting the SOUD sync queue because of a database locked error.

References

Based on database locked errors seen in #11816 (comment)

Addresses the general case seen in this error traceback from the above:

ERROR 2024-02-14 13:38:00,638 kolibri.core.tasks.job Job 50 raised an exception: Traceback (most recent call last):
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/backends/utils.py", line 64, in execute
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/tasks/job.py", line 329, in execute
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/tasks/registry.py", line 238, in __call__
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/auth/tasks.py", line 433, in soud_sync_processing
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/device/soud.py", line 360, in execute_syncs
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/device/soud.py", line 264, in request_sync
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 807, in save
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 846, in save_base
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/dispatch/dispatcher.py", line 191, in send
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/dispatch/dispatcher.py", line 192, in <listcomp>
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/device/signals.py", line 24, in sync_queue_save_update_user_sync_status
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/device/models.py", line 554, in update_status
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/query.py", line 489, in update_or_create
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 807, in save
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 838, in save_base
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 904, in _save_table
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/base.py", line 955, in _do_update
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/query.py", line 667, in _update
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/sql/compiler.py", line 1204, in execute_sql
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/sql/compiler.py", line 899, in execute_sql
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/models/sql/compiler.py", line 889, in execute_sql
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/backends/utils.py", line 64, in execute
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/utils.py", line 94, in __exit__
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/utils/six.py", line 685, in reraise
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/backends/utils.py", line 64, in execute
  File "/home/runner/work/kolibri/kolibri/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
django.db.utils.OperationalError: database is locked

Reviewer guidance

Could this have any unexpected negative side effects? This feels like the right thing to do, as we already use exactly the same decoration mechanism to protect sync queue update or create from database locked errors in the sync hooks.

---

Testing checklist

• Contributor has fully tested the PR manually
• If there are any front-end changes, before/after screenshots are included
• Critical user journeys are covered by Gherkin stories
• Critical and brittle code paths are covered by unit tests

PR process

• PR has the correct target branch and milestone
• PR has 'needs review' or 'work-in-progress' label
• If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
• If this is an important user-facing change, PR or related issue has a 'changelog' label
• If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

• Automated test coverage is satisfactory
• PR is fully functional
• PR has been tested for accessibility regressions
• External dependency files were updated if necessary (yarn and pip)
• Documentation is updated
• Contributor is in AUTHORS.md

[github-actions]

Build Artifacts

Asset type	Download link
PEX file	kolibri-0.16.0rc0.dev0_git.4.g8a8ffcfa.pex
Windows Installer (EXE)	kolibri-0.16.0rc0.dev0+git.4.g8a8ffcfa-unsigned.exe
Debian Package	kolibri_0.16.0rc0.dev0+git.4.g8a8ffcfa-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.0rc0.dev0+git.4.g8a8ffcfa-0.4.0.dmg
Android Package (APK)	kolibri-0.16.0rc0.dev0+git.4.g8a8ffcfa-0.1.0-debug.apk
TAR file	kolibri-0.16.0rc0.dev0+git.4.g8a8ffcfa.tar.gz
WHL file	kolibri-0.16.0rc0.dev0+git.4.g8a8ffcfa-py2.py3-none-any.whl