pytest-aiohttp freezes when using asyncio.create_subprocess_exec in a process that reads from stdin

[pablogsal]

Long story short

pytest-aiohttp freezes when using asyncio.create_subprocess_exec to create a process that reads from stdin using a pipe.

Expected behaviour

The test suite does not hang when executing tests that use asyncio.create_subprocess_exec.

Actual behaviour

The test suite using pytest-aiohttp hangs if two or more test use asyncio.create_subprocess_exec reading from stdin.

Steps to reproduce

Create a file called test.py with this code

import asyncio

async def test_1():
    process = await asyncio.create_subprocess_exec(
            "cat",
            stderr=asyncio.subprocess.PIPE,
            stdin=asyncio.subprocess.PIPE,
            stdout=asyncio.subprocess.PIPE
            )
    stdout, stderr = await process.communicate(input="asyncio rules".encode())

    assert stdout == b"asyncio rules"

async def test_2():
    process = await asyncio.create_subprocess_exec(
            "cat",
            stderr=asyncio.subprocess.PIPE,
            stdin=asyncio.subprocess.PIPE,
            stdout=asyncio.subprocess.PIPE
            )
    stdout, stderr = await process.communicate(input="asyncio rules".encode())

    assert stdout == b"asyncio rules"

and then run python -m pytest test.py. The interpreter hangs when running the second test.

Your environment

Installed packages:

aiohttp==3.4.4
async-timeout==3.0.1
atomicwrites==1.2.1
attrs==18.2.0
chardet==3.0.4
idna==2.7
more-itertools==4.3.0
multidict==4.4.2
pluggy==0.8.0
py==1.7.0
pytest==3.9.2
pytest-aiohttp==0.3.0
six==1.11.0
yarl==1.2.6

I was able to reproduce this issue in MacOs and Linux:

uname -a
Darwin C02VL073HTDG 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

uname -a
Linux RedHat 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[pablogsal]

Git-bisecting with my reproducer points to commit 5853890 as the first bad commit.

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #1600 (aiohttp for long data processing), #2956 (Enable logging when aiohttp.web is used as a program), #2254 (when use aiodns), #2963 (aiohttp doesn't use system settings), and #2729 (400 milliseconds delay using aiohttp).

[asvetlov]

I suspect that something is bad with adding child watcher here: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/test_utils.py#L436-L441

[pablogsal]

I suspect that something is bad with adding child watcher here: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/test_utils.py#L436-L441

Indeed, removing those lines fixes the problem.

[Dreamsorcerer]

I believe the related code here has moved to pytest-asyncio, so if it's still an issue it likely needs to be reported against that project now.