Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid the multiprocessing forkserver method in Python 3.14 #3442

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

befeleme
Copy link

Pull Request check-list

Please make sure to review and check all of these items:

  • Do tests and lints pass with this change?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

This change is essentially a copy of: giampaolo/pyftpdlib#656

Python 3.14 has changed the default multiprocessing method from fork to forkserver which causes failures like this one:

____________________ TestMultiprocessing.test_redis_client _____________________

self = <tests.test_multiprocessing.TestMultiprocessing object at 0x7f646b43d250>
r = <redis.client.Redis(<redis.connection.ConnectionPool(<redis.connection.Connection(host=localhost,port=6379,db=0)>)>)>

    def test_redis_client(self, r):
        "A redis client created in a parent can also be used in a child"
        assert r.ping() is True
    
        def target(client):
            assert client.ping() is True
            del client
    
        proc = multiprocessing.Process(target=target, args=(r,))
>       proc.start()

tests/test_multiprocessing.py:163: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/usr/lib64/python3.14/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <Process name='Process-9' parent=996 initial>
file = <_io.BytesIO object at 0x7f646bbee390>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle local object <function TestMultiprocessing.test_redis_client.<locals>.target at 0x7f646bb0f530>
E       when serializing dict item '_target'
E       when serializing multiprocessing.context.Process state
E       when serializing multiprocessing.context.Process object

I verified this makes tests pass with Python 3.14.0a2 in Fedora's build environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants