Subprocess launched in Uvicorn worker always returns zero exit code

[julioasotodv]

Hi everyone, I am facing one of those little bugs that almost seem impossible to happen (at least IMHO):

Imagine that I have the following "ASGI server app" (server.py), in which I want to make a call to a bash command using subprocess.Popen (I know, there is no ASGI app, but this is just an example):

import subprocess

# The command mkdir without any other
# arguments would normally return nonzero 
# exit code (it returns 1):
proc = subprocess.Popen(["mkdir"], stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)

output, error = (str(s.decode()).strip() for s in proc.communicate())

print("Output: ", output)
print("Error: ", error)

# However, inside a Uvicorn worker, it returns 0 exit code:
print("Exit code: ", proc.returncode)

And we run our "app" with:

gunicorn -k uvicorn.workers.UvicornWorker server.py

The exit code result of subprocess.Popen is always 0, even though it shouldn't. The output is:

Output:
Error: mkdir: missing operand
Try 'mkdir --help' for more information.
Exit code: 0

However, if I just use Gunicorn's default sync workers:

gunicorn server.py

The return code is the right one:

Output:
Error: mkdir: missing operand
Try 'mkdir --help' for more information.
Exit code: 1

Is there any reason why Uvicorn's Gunicorn worker always return zero exit code from subprocess.Popen?

I know it is a weird thing to do, but I actually need a bash call inside the Uvicorn workers.

Thank you!

[euri10]

seems to work fine with the following
note that subprocess will be blocking
your example is as you said weird and franckly I'm dubious you could even launch the gunicorn part that you said was working, at least for me it told me I had no app was declared.

import asyncio

import uvicorn
from fastapi import FastAPI

app = FastAPI()


@app.get("/mkdir")
async def sp():
    proc = await asyncio.create_subprocess_shell(
        "mkdir",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)
    stdout, stderr = await proc.communicate()
    return {"stdout": stdout, "stderr": stderr}


if __name__ == '__main__':
    uvicorn.run("sp:app", reload=True)

{
  "stdout": "",
  "stderr": "mkdir: missing operand\nTry 'mkdir --help' for more information.\n"
}

[julioasotodv]

No, I wasn’t able to launch the ASGI app because I didn’t create it, but that wasn’t the point.

The point is that the proc.returncode with the uvicorn worker is wrong, as it always returns zero even if the command is erroneous

[euri10]

Taking aside this makes no sense to do such things, what you experience I think is a misuse of subprocess, should you use the higher level api run it displays 1 in both cases

import subprocess

# The command mkdir without any other
# arguments would normally return nonzero
# exit code (it returns 1):
proc = subprocess.run(["mkdir"])

# output, error = (str(s.decode()).strip() for s in proc.communicate())

print("Output: ", proc.stdout)
print("Error: ", proc.stderr)

# However, inside a Uvicorn worker, it returns 0 exit code:
print("Exit code: ", proc.returncode)

[julioasotodv]

It turns out that quite a bunch of libraries perform that call by using the autover library: https://github.com/pyviz-dev/autover

If Popen does not work correctly, uvicorn turns out to be incompatible with such libraries, for instance.

[julioasotodv]

@euri10 also, in the example you posted above with subprocess.run() it only works (returning the correct returncode) since your stdout and stderr are nor redirected with subprocess.PIPE back to the Python process. That is why the proc.stdout and proc.stderr turn out to be None.

I believe that Uvicorn workers do not handle PIPEs right... Even though I have no idea why.

[julioasotodv]

I have tried Gunicorn with other async worker (aiohttp's) and the exact same bug arises. I believe it happens because of the combination of os.fork() that Gunicorn performs + async code in the forked process...

I believe this is not Uvicorn's fault, so I will close this issue for now. Sorry for the inconvenience.

[weiyiyin0321]

I have tried Gunicorn with other async worker (aiohttp's) and the exact same bug arises. I believe it happens because of the combination of os.fork() that Gunicorn performs + async code in the forked process...

I believe this is not Uvicorn's fault, so I will close this issue for now. Sorry for the inconvenience.

@julioasotodv
Did you ever resolve your issue? I'm running into the same problem.

[florian-sattler]

For everybody having the same issue, I solved the problem by using shell=True.
Note: The command now has to be a string and not a list anymore.

proc = subprocess.Popen("mkdir", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

[euri10]

See also #895