io.IOBase destructor silence I/O error on close() by default #62948
Comments
|
Running the file couple of times will make the interpreter fail with: libgcc_s.so.1 must be installed for pthread_cancel to work From what I've seen it is triggered from PyThread_delete_key (tries to load libgcc_s.so at that time). How does it happen? The main thread will close fd 4 (because fh object is getting dereferenced to 0) exactly at the same time libpthread will try to open and read libgcc_s.so with the same descriptor (4) It's fairly obvious that the file handling in bug.py is a bug, but the interpreter should not crash like that ! This doesn't happen on python2.7. Also, python2.7 appears to be linked with libgcc_s.so.1 directly while the 3.x does not (I've tried 3.2 from ubuntu repos, and built 3.3 and 3.4 myself on ubuntu 12.04.2) - at least that's what ldd indicates. |
ionelmc
mannequin
added
build
|
Unfortunately, there's not much we can do about it: if dlsym() fails - which is the case here either because read() fails with EBADF, or because the file descriptor now points to another stream (i.e. not libgcc), the libc aborts (here upon pthread_exit(), not PyThread_delete_key()): Program received signal SIGABRT, Aborted. So if you're unlucky and end up closing the FD referring to libgcc used by dlopen/dlsym, you're pretty much screwed, and there's no way around this. Note that you specific problem (upon PyThread_delete_key()) doesn't occur for python 2.7 because it uses and ad-hoc TLS implementation, whereas Python 3 uses the platform native TLS. Unfortunately, we can't do much against this, so I'm tempted to close this as "wont fix". |
|
Perhaps the only thing we could do would be try to "preload" libgcc by calling one of those APIs at startup? But I'm not sure it's a good idea to add such a platform-specific hack (for what is arguably an obscure and rare issue). |
|
Maries Ionel Cristian> ubuntu 12.04.2 What is the version of your libc library? Try something like "dpkg -l libc6". -- Could this issue be related to this glibc issue? I ran your script on Python 3.4 in a shell loop: I'm unable to reproduce the issue. I'm running Fedora 18 with the glibc 2.16. |
|
Oh ok, I'm able to reproduce the issue with the system Python 3.3: $ while true; do echo "loop"; python3.3 bug.py || break; done
loop
...
loop
libgcc_s.so.1 must be installed for pthread_cancel to work
Abandon (core dumped) |
Yeah, I was thinking about doing this in PyThread_init_thread() but...
then thought the exact same thing. Also, this exact problem can possibly show up anywhere dlopen() is |
2.15-0ubuntu10.4 I don't think it's that obscure ... uwsgi has this issue |
|
2013/8/17 Maries Ionel Cristian <report@bugs.python.org>:
The error message is maybe the same, the reason is completly different: The root cause is an arbitrary limit on the size of the virtual memory. Here the bug is that a thread B closes a file descriptor opened in a |
|
Well anyway, is there any way to preload libgcc ? Because in python2.x it wasn't loaded at runtime. |
|
2013/8/18 Maries Ionel Cristian <report@bugs.python.org>:
On Linux, you can try to set the LD_PRELOAD environment variable as a LD_PRELOAD=libgcc_s.so.1 python bug.pyYou may need to specify the full path. |
|
Alright ... would it be a very big hack to preload libgcc in the thread module (at import time) ? There is platform specific code there anyway, it wouldn't be such a big deal would it? |
I don't think that'll work.
Yes it was. As explained above, you can get the very same crash upon
IMO yes, but if someone writes a patch, I won't oppose to it :-) |
|
I think this is the same issue I'm getting. I'm building on a 56 core system so that might make me more likely to hit the issue. |
|
Looks like I was mistaken. My cross compiler was trying to load libgcc_s.so.1 from the standard location and not liking the one it found. Fixed for now by setting LD_LIBRARY_PATH to point at the dir containing the right libgcc_s.so.1 |
|
I meant my cross compiled python, not my cross compiler. |
|
I have observed this problem in a production application using Python 3.5 and 3.6 (system-packaged interpreters from Ubuntu) and I would like to mention that this is an effective workaround: provided you do it before creating any threads, and the variable |
|
I looked at this old issue. First of all, "bug.py" filename is accurate: this program contains a bug, but it's non-obvious.
Note: the Python file object remains alive until the "t" local variable is cleared, since "t.fh" contains a strong reference to the file object. Again: it's a bug in the program. "libgcc_s.so.1 must be installed for pthread_cancel to work" is just *one* example of bad thing that can happen. What can be do on the Python side? Well, stop to ignore silently I/O errors when a file is closed by its destructor. I wrote a shy PR 12786 to only log these exceptions in development mode (-X dev). I'm not sure if we could stop to silence these exceptions *by default*. Python has been modified last years to first raise an exception on I/O errors when file.close() is called explicitly, and then a similar change has been done for call to socket.socket.close(). |
vstinner
changed the title
|
To debug remaining "Exception ignored in:" issues, I'm using the following patch: diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index 972a4658b1..be38af3daa 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -710,6 +710,7 @@ class TestCase(object):
# clear the outcome, no more needed
self._outcome = None
+ import gc; gc.collect()
def doCleanups(self):
"""Execute all cleanup functions. Normally called for you afterAnd I run: ./python -X dev -u -m test test_io -v 2>&1|tee log |
|
I wrote PR 12805 to silence IOBase destructor exceptions in test_io. |
|
Is there someone interested to debug remaining "Exception ignored:" logs in test_urllib? test_invalid_redirect (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24be0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok
test_read_bogus (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24cd0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok
test_redirect_limit_independent (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24cd0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24c30>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24460>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24a50>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f247d0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52ea95f0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52ea99b0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2f820>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2fbe0>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2fd70>
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
super().close() # set "closed" flag
File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
okIt's unclear to be if it's a bug in http.client, a bug in the test... or something else. |
|
Maybe the fact that close() exceptions are ignored silently in io.IOBase constructor should be better documented, but I'm not sure where it should be documented. If someone has an idea, please go ahead and write a pull request :-) |
|
The ValueError warnings in test_urllib noted in msg340059 feels like an issue with the test where FakeSocket calls close by itself once it's internal io_refs counter is 0 and during destructor again close is called on an already closed object causing the ValueError.
Maybe a check could be added during flush to make sure the object is not closed by testing for self.closed but I am not sure if closed attribute is guaranteed to be present. Removing the manual call to close in fakesocket could help since the destructor should be taking care of it and I could see no test failures or warnings removing the close as io_refs gets to 0. [1] cpython/Lib/test/test_urllib.py Line 61 in be6dbfb [2] cpython/Lib/test/test_urllib.py Line 67 in be6dbfb [3] Line 919 in be6dbfb [4] Line 2145 in be6dbfb |
|
I was wrong so removing the close call when io_refs gets to zero causes error on test_urllib2 [0] where close is asserted to be True and I only tested test_urllib initially. One another way would be to check for fp.closed in http.client before flushing but as mentioned before I am not sure of the attribute being always present and also to modify http.client code just for tests. [0] cpython/Lib/test/test_urllib2.py Line 1685 in f12ba7c |
|
I created bpo-37223: test_io logs Exception ignored in: <function IOBase.__del__ at 0x7f2f3c73e4b0> warnings. |
|
I closed the issue. I consider that the initial issue has been fixed and all related issues have been fixed as well:
-- For Python 3.7, there is no known workaround. It seems like Python 2.7 in debug mode logs an error: $ python2.7-dbg bug.py
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor |
|
Same problem using Python 3.6.8 on Ubuntu 18.04 LTS. For now, solve this using LD_PRELOAD=libgcc_s.so.1 python3 ...For more details and pocs: https://github.com/WHK102/wss/issues/2 |
To be clear: this issue is NOT a bug in Python, but a bug in your application. You must fix your application. You can try to run it on Python 3.8 with python3.8 -X dev to get a log on the error. Good luck to debug it ;-) The change that I made is only to add a log to help developers to find their own bugs ;-) |
I don't think that's entirely true. I think CPython needs to be linked against libgcc_s.so, so that this class of application bugs will no longer manifest as interpreter crashes. I filed bpo-37395 for that. |
|
I hope that when an error occurs, python tells me what the problem is. The abort core error is a problem at a lower level than python because python is not able to recognize or handle the error. The main problem is that I exceeded the maximum number of process threads supported by the kernel. What I hope is that python throws an exception when it exceeds this limit or when it cannot access the pointer in memory of the process thread. The problem is not if the script is right or wrong, but that Python is not able to recognize and handle the problem. A generic message saying that an error occurred without indicating where and how it occurred is a python bug. |
See also bpo-44434: "_thread module: Remove redundant PyThread_exit_thread() call to avoid glibc fatal error: libgcc_s.so.1 must be installed for pthread_cancel to work". |
A reference loop was resulting in the `fileobj` held by the `GzipFile` being closed before the `GzipFile`. The issue started with pythongh-89550 in 3.12, but was hidden in most cases until 3.13 when pythongh-62948 made it more visible.
A reference loop was resulting in the `fileobj` held by the `GzipFile` being closed before the `GzipFile`. The issue started with pythongh-89550 in 3.12, but was hidden in most cases until 3.13 when pythongh-62948 made it more visible. (cherry picked from commit 7f39137) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
A reference loop was resulting in the `fileobj` held by the `GzipFile` being closed before the `GzipFile`. The issue started with pythongh-89550 in 3.12, but was hidden in most cases until 3.13 when pythongh-62948 made it more visible. (cherry picked from commit 7f39137) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
…130670) gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) A reference loop was resulting in the `fileobj` held by the `GzipFile` being closed before the `GzipFile`. The issue started with gh-89550 in 3.12, but was hidden in most cases until 3.13 when gh-62948 made it more visible. (cherry picked from commit 7f39137) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
…130669) gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) A reference loop was resulting in the `fileobj` held by the `GzipFile` being closed before the `GzipFile`. The issue started with gh-89550 in 3.12, but was hidden in most cases until 3.13 when gh-62948 made it more visible. (cherry picked from commit 7f39137) Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: