gh-100228: Warn from os.fork() if other threads exist.

[gpshead]

Not comprehensive, best effort warning. There are cases when threads exist that this code currently does not detect.

Starting with a DeprecationWarning for now, it is less disruptive than something like RuntimeWarning and most likely to only be seen in people's CI tests - a good place to start with this messaging.

TODO:

• doctests for multiprocessing failed with an assertion crash in 2854473 about the error being set unexpectedly on one later commit, why? did I fix that? I need to figure out how to run those and try to reproduce it. (could not reproduce, no way it could be related seen upon code inspection)

• Issue: raise a Warning when os.fork() is called and the process has multiple threads #100228

[netlify]

✅ Deploy Preview for python-cpython-preview canceled.

Name	Link
🔨 Latest commit	a91a4d1
🔍 Latest deploy log	https://app.netlify.com/sites/python-cpython-preview/deploys/639aae96df45c9000879b05a

[zzzeek]

is the use of fork() with a threaded parent going to be disallowed in a future release? That's what DeprecationWarning means. We have test code which makes use of fork to run some test cases in a separate process. There is no interaction with threads from the parent and the code is safe. We cannot use spawn or forkserver because we are running a Python function defined in the test itself and it cannot be serialized. We can silence this warning but it is not clear here, or even in the changelog given, if this functionality is going to be actually disallowed and I think it's critical to make that clear (also that the answer should be: "no" :) )

[gpshead]

we can't disable it because people depend on it whether that's wise or not, but it is always the wrong thing to do. we're not going to create a "YouWillSufferMysteriousProblemsWarning" for such purposes. DeprecationWarning is accurate because it is something people used to do a lot but underlying platforms have significantly moved on such that things that relied on what they could get away with 15-20 years ago are no longer feasible in many normal environments today.

the deadlocks can come from the platform level and other low level libraries including the C stdlib and malloc and whatnot. it does not matter if you think you interact with the threads at all, just that they exist and are running is enough to lead to scenarios outside your control where problems happen randomly when forking.

for functions defined in the test themselves, the normal advise is to refactor the code so it is something pickle can handle (generally: not being defined on the fly)

[zzzeek]

Where can I get your input on an issue I'm seeing where I dont actually see why this warning is being emitted? This has to do with the use of the pytest-xdist plugin which uses a tool called execnet. When using this plugin, if our own test uses fork, we get the warning. however, I've created a suite that runs through threading.enumerate() in the parent process, and in the child process - there is no additional thread spawned in either, there is only a single thread, the "main" thread.

output can be seen as follows - as you can see I'm not able to find any additional threads that are known to the Python interpreter. So....execnet is spawning threads outside of the interpreter? is that what has to be happening? is that a bug for them?

initialized: 1/1 workerINFO:main:THREAD FROM MAIN PROCESS 14974: <_MainThread(MainThread, started 140612028988288)>

1 worker [1 item]      
INFO:main:THREAD FROM CHILD PROCESS 14975 (parent: 14974): <_MainThread(MainThread, started 139972663720832)>

.
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=14975) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 10.17s =====================================================================================

I've looked at execnet a few times and so far it's opaque to me what it's doing or how pytest-xdist uses it.

[zzzeek]

i've opened an issue at pytest-dev/pytest-xdist#1186