-
-
Notifications
You must be signed in to change notification settings - Fork 724
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Subprocess returncode is not detected when running Gunicorn with Uvicorn (with fix PR companion) #894
Closed
2 tasks done
Labels
Comments
Was able to reproduce with a Starlette app as well. import subprocess
from starlette.applications import Starlette
from starlette.routing import Route
from starlette.responses import JSONResponse
async def run_subprocess(request):
result = subprocess.run(
["python", "-c", "import sys; sys.exit(1)"], capture_output=True
)
return JSONResponse({"returncode": result.returncode})
app = Starlette(routes=[Route("/run", run_subprocess, methods=["POST"])]) With plain Uvicorn: always returns $ uvicorn app:app
$ curl -X POST localhost:8000/run
{"returncode":1} With Gunicorn: always getting $ gunicorn -k uvicorn.workers.UvicornWorker app:app
$ curl -X POST localhost:8000/run
{"returncode":0} When running off from #895, both cases always return |
Awesome! Thanks @florimondmanca ! 🚀 ☕ |
Closed
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
master
.Describe the bug
When starting Gunicorn with Uvicorn worker(s), if the app uses
subprocess
to start other processes and captures the output, theirreturncode
is in most cases0
, even if the actual exit code was1
.To reproduce
Take this minimal FastAPI app (or replace with Starlette),
main.py
:Then run it with:
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Open the browser at http:127.0.0.1:8000/docs and send a request to
/run
.Expected behavior
The detected
returncode
should always be1
, as the subprocess always exits with1
.Actual behavior
In most of the cases it will return a
returncode
of0
. Strangely enough, in some cases, it will return areturncode
of1
.Debugging material
This is because the
UvicornWorker
, which inherits from the base Gunicorn worker, declares a methodinit_signals()
(overriding the parent method) but doesn't do anything. I suspect it's because the signal handlers are declared in theServer.install_signal_handlers()
with compatibility withasyncio
.But the
UvicornWorker
process is started withos.fork()
by Gunicorn (if I understand correctly) and by the point it is forked, the Gunicorn "Arbiter" class (that handles worker processes) already set its own signal handlers.And the signal handlers in the Gunicorn base worker reset those handlers, but the
UvicornWorker
doesn't. So, when a process started withsubprocessing
is terminated, theSIGCHLD
signal is handled by the GunicornArbiter
(as if the terminated process was a worker) instead of by theUvicornWorker
.Disclaimer: why the
SIGCHLD
signal handling in the GunicornArbiter
alters thereturncode
of a process run withsubprocess
, when capturing output, is still a mystery to me. But I realized the signal handler in theArbiter
is expected to handle dead worker processes. And worker subclasses all seem to reset the signal handlers to revert those signals set by theArbiter
.I'm also submitting a PR to fix this: #895. It's just 3 lines of code. But debugging it and finding it took me almost a week. 😅
Environment
uvicorn --version
:Running uvicorn 0.13.1 with CPython 3.8.5 on Linux
(it's actually installed from source, for debugging)gunicorn (version 20.0.4)
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Additional context
I'm pretty sure this issue #584 is related to the same problem.
The text was updated successfully, but these errors were encountered: