fix: run Celery tasks asynchronously in dev mode

In `tutor dev`, Celery tasks were running eagerly (in-process) rather than being delegated to the worker containers for
asynchronous processing. This difference in behavior between
`tutor dev` and `tutor local` meant that certain code paths
and race-condition bugs would not surface when testing in dev.

The cause is that CELERY_ALWAYS_EAGER is set to True in the
upstream devstack settings file. That's because Devstack does
not support async LMS/CMS workers; it runs all those tasks
in-process.

The fix is to simply override CELERY_ALWAYS_EAGER to False.
We do this in the common settings file for simplicity, but the
change should not affect production mode, which has always
had CELERY_ALWAYS_EAGER equal False.

For unit tests, we override CELERY_ALAYS_EAGER back to True,
because many of them are written to assume in-process tasks.

changelog.d/20231025_122158_kyle_celery_async.md

@@ -0,0 +1 @@
+- [Bugfix] In development mode, Celery tasks will now be delegated to worker containers for asynchronous execution rather than eagerly exected by the lms/cms app processes. This matches the behavior of production mode and should enable more robust testing & debugging of Celery tasks. Unit tests, however, will still eagerly execute Celery tasks in-process (by @kdmccormick). 

tutor/templates/apps/openedx/settings/partials/common_all.py

@@ -235,5 +235,8 @@
     "user": None,
 }
 
+# Run Celery tasks asynchronously on the workers (rather than blocking the app process).
+CELERY_ALWAYS_EAGER = False
+
 {{ patch("openedx-common-settings") }}
 ######## End of settings common to LMS and CMS

tutor/templates/apps/openedx/settings/partials/common_test.py

@@ -1,3 +1,7 @@
 # Fix MongoDb connection credentials
 DOC_STORE_CONFIG["user"] = None
 DOC_STORE_CONFIG["password"] = None
+
+# In edx-platform CI, unit tests are run in-process, so many of them break if run
+# async on the workers. So, we default to testing them in-process as well.
+CELERY_ALWAYS_EAGER = True