Bad file descriptor problem still exists as of today with uvicorn workers #3013
Comments
crankedguy
changed the title
|
To me, what I read here is: "I don't want to read all the issues, can you summarize for me?". As I mentioned on #1877 (comment), I believe the problem is on I've also asked about the So... I've already offered, and dedicated too much of my time. Also, I find it very rude of you for pinging me here. 👍 |
|
Actually I did read it, but at some point I got simply lost. There are countless threads involved, changes implemented, reverted.. I also saw that this was more or less ignored on this side sadly. As fas as I understood the SIGQUIT handler was implemented in uvicorn worker but the problem persists... On being rude : Although I can't really see a rudeness in a simple question, it was certainly not the intention if you consider it like that. So you can take my sorry for it. I won't bother any further... |
🤷♂️
Apologize accepted.
The problem that was solved is the shutdown event not running when using uvicorn, which was indeed solved. The compromise was having the message "Bad file descriptor" more often on the logs. The problem that needs to be solved is mentioned here: #1877 (comment). |
Yes I also saw that one. That comment came after that particular issue was just closed, and there was no activity on it except from users still having the problem 🙏 Sometimes you only get the already documented and when you get - also already documented - EDIT full trace : |
|
@Kludex you weren't ignored. I offered a response to your issue that didn't match your expectation at this time. This happen sometimes, but don't say it was ignored... @crankedguy that's the good approach. All issues closed are closed and if a new problem arise or if an issue still exist a new ticket should be created. This is described in the contributing doc. About this issue. I didn't have yet re-tested latest master with uvicorn. This is planned this we. I have been side tracked with some coming changes in the http parser and some other changes for the new release. The behavior is not reproduced with others workers, The expected behaviour is the following: A worker should expect to receive 2 signals for termination : SIGTERM and SIGQUIT. This allows graceful shutdown. This message is sent by the arbiter. The arbiter control the worker and do not simply pass signals. Anyway I will try to understand what is the design difference there that causes this issue. |
|
That would be great because this is kinda a problem not only with command line invoke instances but also if you handle them over systemd which leads to unclean unit shutdowns |
|
What worked for me if watchdog and socketio was being used in the application (in this case flask) the error would show up as socket is also a file in way. Hopefully this is useful and it helps |
|
In this comment, @gg349 very helpfully tracked down the uvicorn commit that re-ordered calls to close the asyncio servers and close the sockets. In the asyncio documentation for loop.create_server, it says:
That means that There is nothing gunicorn can do for this issue. The error happens in the worker, where uvicorn is closing the socket after it was already closed. I suggest we close the issue here if this diagnosis makes sense to @crankedguy and @Kludex. |
|
Note that this only happens with the uvloop dependency. Python sockets can actually be closed multiple times without issue. Python tracks when a socket has been closed and only issues a close to the OS once. However, if you close a file descriptor out from underneath the Python socket, then closing it from Python will cause an EBADF error: >>> s = socket.socket(socket.SOCK_STREAM)
>>> socket.close(s.fileno())
>>> s.close()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.11/socket.py", line 503, in close
self._real_close()
File "/usr/lib/python3.11/socket.py", line 497, in _real_close
_ss.close(self)
OSError: [Errno 9] Bad file descriptorGunicorn actually passes a So, it doesn't normally cause problems that uvicorn closes both the server and the socket. However, things get weird with uvloop and gunicorn. When the server gets closed, uvloop calls
In other words, socket handles in uvloop expect that the base handle has already closed the fd and want to simply call Gunicorn could add a defensive |
|
If uvicorn cannot track which sockets it has and has not passed to servers, it may want to have its own defensive If we consider any change in gunicorn itself, I'd consider trying to remove the socket wrapper classes, or making them actually inherit from |
Refactor socket creation to remove the socket wrapper classes so that these objects have less surprising behavior when used in worker hooks, worker classes, and custom applications. Close #3013.
|
I've opened a draft PR for getting rid of the socket wrapper classes and verified that this fixes this issue with uvloop. |
Refactor socket creation to remove the socket wrapper classes so that these objects have less surprising behavior when used in worker hooks, worker classes, and custom applications. Close #3013.
Refactor socket creation to remove the socket wrapper classes so that these objects have less surprising behavior when used in worker hooks, worker classes, and custom applications. Close #3013.
|
thanks @tilgovi Waiting for you PR to be ready |
Refactor socket creation to remove the socket wrapper classes so that these objects have less surprising behavior when used in worker hooks, worker classes, and custom applications. Close benoitc#3013.
commit e4b5d48 Merge: a24ff07 ec52843 Author: Benoit Chesneau <bchesneau@gmail.com> Date: Sat Aug 10 10:23:20 2024 +0200 Merge pull request benoitc#3188 from tnusser/patch-1 Update sock.py by making bind list creation more readable commit ec52843 Author: Tobias Nusser <tobiasnusser@web.de> Date: Wed Apr 17 12:48:26 2024 +0200 Update sock.py by making bind list creation more readable Implementing suggestion by reviewer benoitc#3127 (comment) to make code more readable. commit a24ff07 Author: Randall Leeds <randall@bleeds.info> Date: Thu Dec 28 00:57:50 2023 -0800 Use plain socket objects instead of wrapper classes Refactor socket creation to remove the socket wrapper classes so that these objects have less surprising behavior when used in worker hooks, worker classes, and custom applications. Close benoitc#3013.
Hello,
why was this
#1877
closed and further comments after closure completely ignored?
I must admit the crosslinking between the project issues and the conversation is extremely hard to follow, for me honestly not at all anymore now what is done, reverted, has to be done, etc.
encode/uvicorn#1710
Can someone please shed some light onto if and how this will be fixed, because this is still an issue as of today with gunicorn 20.1.- and uvicorn / uvicornworkers 0.22, and is also very easy to reproduce every single time the server is shutdown (e.g. with Ctrl-C)
Maybe
@Kludex ??
Answer highly appreciated, thanks
The text was updated successfully, but these errors were encountered: