Deadlock when mixing event loops and subprocesses

[ehrmann]

BPO	43890
Nosy	@ehrmann
Files	subprocess_asyncio_deadlock.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-19.18:36:19.433>
labels = ['type-bug', '3.8']
title = 'Deadlock when mixing event loops and subprocesses'
updated_at = <Date 2021-04-19.18:36:19.433>
user = 'https://github.com/ehrmann'

bugs.python.org fields:

activity = <Date 2021-04-19.18:36:19.433>
actor = 'thomas'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-19.18:36:19.433>
creator = 'thomas'
dependencies = []
files = ['49967']
hgrepos = []
issue_num = 43890
keywords = []
message_count = 1.0
messages = ['391386']
nosy_count = 1.0
nosy_names = ['thomas']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43890'
versions = ['Python 3.8']

[ehrmann]

When mixing code that spawns subprocesses with code that creates event loops, Python appears to deadlock.

In the attached example, when WORKERS = 16 and ASYNC_WORKERS = 8, Python will sometimes (50% of the time?) deadlock, never exiting, with no exceptions raised. Oddly, if ASYNC_WORKERS is set to 0 or 16 (i.e., only subprocesses or only event loops), it runs reliably.

I tested this in an x86_64 Ubuntu 20.04.2 VM with Python 3.8.6. I was only able to reproduce it when the VM had more than one CPU, and I was unable to reproduce it on x86_64 MacOS.

[kumaraditya303]

Reproduced on 3.11, I think it is because of fork

[kumaraditya303]

Duplicate of #90368

So after some investigation, this issue is a duplicate of #90368 and if you remove the print calls it is fixed. Moreover the fix in #30310 fixes this issue so closing as duplicate.