Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prefs Multiprocessing.Manager can't boot in ipython/interactive python sessions? #128

Closed
sneakers-the-rat opened this issue Aug 28, 2021 · 1 comment
Labels

Comments

@sneakers-the-rat
Copy link
Contributor

Can't even import from ipython because the mp.Manager doesn't boot, have had various errors over time, often it's that the socket it relies on closes prematurely, this one is slightly different but this is a reminder to fix it.

I think we should just warn fallback to using a standard dictionary if the multiprocess access fails, for 99% of use cases this will be fine (prefs shouldn't dynamically change that much during operation, there are some circumstances where it might, eg. programmatic preference setting, and different processes might get out of sync, this should be pretty rare).

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 264, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/runpy.py", line 234, in _get_code_from_file
    with io.open_code(decoded_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/masking/my/directory/structure/<input>'
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/masking/my/directory/structure/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/masking/my/directory/structure/lib/python3.8/site-packages/autopilot/__init__.py", line 4, in <module>
    from autopilot.setup import setup_autopilot
  File "/masking/my/directory/structure/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/masking/my/directory/structure/lib/python3.8/site-packages/autopilot/setup/setup_autopilot.py", line 19, in <module>
    from autopilot.setup.forms import Autopilot_Setup, DIRECTORY_STRUCTURE
  File "/masking/my/directory/structure/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/masking/my/directory/structure/lib/python3.8/site-packages/autopilot/setup/forms.py", line 10, in <module>
    from autopilot.prefs import _DEFAULTS, Scopes
  File "/masking/my/directory/structure/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "//masking/my/directory/structure/lib/python3.8/site-packages/autopilot/prefs.py", line 111, in <module>
    _PREF_MANAGER = mp.Manager() # type: mp.managers.SyncManager
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/Users/jonny/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
@sneakers-the-rat
Copy link
Contributor Author

The prefs class now uses the same check that multiprocessing does to check if it can be started in this context:
https://github.com/wehr-lab/autopilot/blob/b56a3002ad1a500f1ebe7d8291ccedd3c592cbd0/autopilot/prefs.py#L112

and has an env setting where if AUTOPILOT_NO_PREFS_MANAGER is set then it uses a python dict instead of one within a multiprocessing manager.

Long term solution is to switch to a file-based model using pydantic (rather than trying to share a live dict between processes, which is almost never necessary and encourages misusing prefs (as i have done all over the place in the library)), see #155

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant