testing.DaphneProcess fails when multiprocessing start method set to spawn

[bjd183]

I'm attempting to upgrade my project from Python 3.7.5 to Python 3.8.1. I'm on MacOS 10.15.3 using Python 3.7.5 and Daphne 2.4.1 with Channels 2.4.0 and Django 3.0.2. The issue manifests while invoking tests using the standard django approach: manage.py test.

The default start method for multiprocessing on MacOS (where I perform my local test) was changed from fork to spawn in Python 3.8 due to issues documented here. The result of this change is that Python attempts to pickle DaphneProcess in order to spawn. This is true in Python 3.8 where spawn is the default and in Python 3.7 if the start method is changed from fork to spawn. The first error is due to the presence of non-picklable lambda expressions in the initializer.

Traceback (most recent call last):
File "lib/python3.7/site-packages/django/test/testcases.py", line 267, in call
self._pre_setup()
File "lib/python3.7/site-packages/channels/testing/live.py", line 52, in _pre_setup
self._server_process.start()
File "lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "lib/python3.7/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "lib/python3.7/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'DaphneProcess.init..'

This is easily remedied by replacing lambda: None with type(None) in testing.DaphneProcess.init. The next symptom is that the spawned process doesn't retain any of the Django state from django.setup() and Django warns that apps aren't loaded.

File "", line 1, in
File "lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "lib/python3.7/site-packages/channels/auth.py", line 12, in
from django.contrib.auth.models import AnonymousUser
File "lib/python3.7/site-packages/django/contrib/auth/models.py", line 2, in
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "lib/python3.7/site-packages/django/contrib/auth/base_user.py", line 47, in
class AbstractBaseUser(models.Model):
File "lib/python3.7/site-packages/django/db/models/base.py", line 107, in new
app_config = apps.get_containing_app_config(module)
File "lib/python3.7/site-packages/django/apps/registry.py", line 252, in get_containing_app_config
self.check_apps_ready()
File "lib/python3.7/site-packages/django/apps/registry.py", line 135, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

It seems that because Daphne can be used without Channels/Django, calling django.setup() should be performed outside Daphne but I'm not sure if there is another approach or if I am missing something.

I inserted the following into channels.auth:
import django
if not django.apps.apps.ready:
django.setup()
in an effort to load apps (which worked) but apparently the spawn also discarded environment variables and any information about the current test database resulting in a connection to a database without any applied migrations. Down the rabbit hole I go.

My current workaround is to call `multiprocessing.set_start_method('fork') in my django manage.py file. While this reverts to the same behavior I have by default on Python 3.7.5, I anticipate that 1) there may be hidden issues with this because Channels/Django uses threads to make database calls, and 2) fork may be deprecated in the future on MacOS. based on the python issue referenced above.

At this point it is not clear to me how much of the problem or the solution is daphne vs channels. Perhaps some of both. It occurs to me that the ability to pass environment variables through DaphneProcess may resolve the database connection issue.

[carltongibson]

My current workaround is to call `multiprocessing.set_start_method('fork') in my django manage.py file.

This is not a bad solution for the moment. (It's still working just fine on Python 3.7) But yes...

[bjd183]

After double checking, environment variables are successfully passed to the spawned process (contrary to my statement above). Also, it seems that this is likely an issue on Windows and Linux although I am unable to verify.

[Jayesh-Mahato]

I am facing the same issue of "Can't pickle local object..." in macOS Catalina.

File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'DaphneProcess.__init__.<locals>.<lambda>'

[bjd183]

I seem to have stumbled upon the solution to this problem. Inspawn mode, the application object is serialized/pickled along with all its references to Django. This fails deep in Django (after fixing the lambda serialization issue in the DaphneProcess initializer). The solution is to either pass a string of the form package.module:application to the initializer (as is done on the command line) and then dynamically import or to call get_default_application() only once inside run() which is invoked in the new process. The result is that the application object and all related Django module imports are performed freshly in the new process. This adds a slight delay to startup but can be partially mitigated by not blocking while waiting for the port to be assigned.

PR to follow.

There may be an issue, however, due to Channels passing the application object directly from ChannelsLiveServerTestCase.

[carltongibson]

@bjd183 -- good work investigating! Look forward to the PR. Sounds fun to look at!

[bjd183]

#361

[carltongibson]

Fixed in #440