Some error paths do not set Exceptions in _testexternalinspection #130052

sergey-miryanov · 2025-02-12T20:22:32Z

Bug report

Bug description:

Following @colesbury findings at #130035 (comment), I did a manual inspection of _testexternalinspection and found a few more cases where Exception should be set.

Will sent a PR soon.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

The text was updated successfully, but these errors were encountered:

…exception

…ion (#130053) Co-authored-by: Victor Stinner <vstinner@python.org>

befeleme · 2025-04-10T15:39:39Z

I see the same issue happening with a7 on aarch64 with freethreading-debug build:

test_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_remote_stack_trace) ... python: /builddir/build/BUILD/Python-3.14.0a7/Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.
Fatal Python error: Aborted
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/test_external_inspection.py", line 63 in test_remote_stack_trace
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 660 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/case.py", line 716 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/unittest/runner.py", line 259 in run
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 127 in main
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/runpy.py", line 88 in _run_code
  File "/builddir/build/BUILD/Python-3.14.0a7/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testexternalinspection (total: 2)

sergey-miryanov · 2025-04-10T17:24:49Z

@befeleme Can you rerun this test with following patch?

diff --git a/Lib/test/test_external_inspection.py b/Lib/test/test_external_inspection.py
index 2ab48a4778b..b71f65fed32 100644
--- a/Lib/test/test_external_inspection.py
+++ b/Lib/test/test_external_inspection.py
@@ -60,6 +60,7 @@ def foo():
                 with open(fifo, "r") as fifo_file:
                     response = fifo_file.read()
                 self.assertEqual(response, "ready")
+                print('test_remote_stack', p, p.pid, type(p.pid))
                 stack_trace = get_stack_trace(p.pid)
             except PermissionError:
                 self.skipTest("Insufficient permissions to read the stack trace")

I want to make sure that the get_stack_trace from _testexternalinspection is actually called (I just checked the code again and don't see anything suspicious).

vstinner · 2025-04-15T14:58:38Z

On Fedora Rawhide x86-64, I get the following unrelated errors:

ERROR: test_async_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_remote_stack_trace) (task_factory_variant='asyncio.new_event_loop')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 146, in test_async_remote_stack_trace
    stack_trace = get_async_stack_trace(p.pid)
OSError: [Errno 14] Bad address

======================================================================
ERROR: test_async_staggered_race_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_staggered_race_remote_stack_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 334, in test_async_staggered_race_remote_stack_trace
    stack_trace = get_async_stack_trace(p.pid)
OSError: [Errno 14] Bad address

======================================================================
ERROR: test_self_trace (test.test_external_inspection.TestGetStackTrace.test_self_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.14t/test/test_external_inspection.py", line 357, in test_self_trace
    stack_trace = get_stack_trace(os.getpid())
OSError: [Errno 14] Bad address

On Fedora 41 AArch64 with Python built with ./configure --disable-gil CFLAGS="-O0" && make -j10, the test pass with success: ./python -m test test_external_inspection -v.

vstinner · 2025-04-15T15:17:34Z

Ah, Fedora builds Python with LTO. I recall that test_external_inspection had issues with LTO (on Fedora). I re-ran tests with LTO on x86-64 and AArch64 and I still fail to reproduce the failure.

befeleme · 2025-04-16T09:35:31Z

Testing again with the patch, a similar failure happens on s390x in another test: test_async_gather_remote_stack_trace.
On aarch64 the build was this time successful. :(

0:24:30 load avg: 2.73 [1/1/1] test_external_inspection worker non-zero exit code (Exit code -6 (SIGABRT))
Re-running test_external_inspection in verbose mode (matching: )
test_async_gather_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_gather_remote_stack_trace) ... python: /builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.
Fatal Python error: Aborted
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/test_external_inspection.py", line 272 in test_async_gather_remote_stack_trace
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 660 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/case.py", line 716 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 122 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/suite.py", line 84 in __call__
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/unittest/runner.py", line 259 in run
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 127 in main
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/runpy.py", line 88 in _run_code
  File "/builddir/build/BUILD/python3.14-3.14.0_a7-build/Python-3.14.0a7/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testexternalinspection (total: 2)
1 test failed again:
    test_external_inspection
== Tests result: FAILURE then FAILURE ==

vstinner · 2025-04-16T13:30:18Z

Ok, I reproduced the error on s390x using the Fedora package python3.14-freethreading-debug.s390x.

The error comes from search_map_for_section() at:

402     exit:
403         if (close(fd) != 0) {
404             PyErr_SetFromErrno(PyExc_OSError);
405         }

The problem is that the exit label is called with an exception set, and close(-1) tries to raise a new exception. In this case, the assertion fails.

I'm working on a fix: #132594

* Don't call close() if the file descriptor is negative. * If close() fails, chain the existing exception.

…H-132594) * Don't call close() if the file descriptor is negative. * If close() fails, chain the existing exception. (cherry picked from commit 014c7f9) Co-authored-by: Victor Stinner <vstinner@python.org>

) (#132598) gh-130052: Fix search_map_for_section() error handling (GH-132594) * Don't call close() if the file descriptor is negative. * If close() fails, chain the existing exception. (cherry picked from commit 014c7f9) Co-authored-by: Victor Stinner <vstinner@python.org>

sergey-miryanov added the type-bug An unexpected behavior, bug, or error label Feb 12, 2025

bedevere-app bot mentioned this issue Feb 12, 2025

gh-130052: Fix some exceptions on error paths in _testexternalinspection #130053

Merged

encukou added the extension-modules C modules in the Modules dir label Feb 13, 2025

sergey-miryanov added a commit to sergey-miryanov/cpython that referenced this issue Feb 13, 2025

Merge branch 'main' into pythongh-130052-test-external-inspection-set…

9679c08

…exception

sergey-miryanov added a commit to sergey-miryanov/cpython that referenced this issue Feb 13, 2025

Merge branch 'main' into pythongh-130052-test-external-inspection-set…

4830892

…exception

sergey-miryanov added a commit to sergey-miryanov/cpython that referenced this issue Feb 17, 2025

Merge branch 'main' into pythongh-130052-test-external-inspection-set…

8bf3c22

…exception

vstinner added a commit that referenced this issue Feb 20, 2025

gh-130052: Fix some exceptions on error paths in _testexternalinspect…

69426fc

…ion (#130053) Co-authored-by: Victor Stinner <vstinner@python.org>

vstinner mentioned this issue Feb 24, 2025

Python 3.14.0a5: test_external_inspection fails on freethreading-debug build on s390x Linux #130035

Closed

colesbury marked this as a duplicate of #130035 Feb 25, 2025

sergey-miryanov closed this as completed Mar 25, 2025

vstinner added a commit to vstinner/cpython that referenced this issue Apr 16, 2025

pythongh-130052: Fix search_map_for_section() error handling

8c71d19

* Don't call close() if the file descriptor is negative. * If close() fails, chain the existing exception.

bedevere-app bot mentioned this issue Apr 16, 2025

gh-130052: Fix search_map_for_section() error handling #132594

Merged

vstinner added a commit that referenced this issue Apr 16, 2025

gh-130052: Fix search_map_for_section() error handling (#132594)

014c7f9

* Don't call close() if the file descriptor is negative. * If close() fails, chain the existing exception.

bedevere-app bot mentioned this issue Apr 16, 2025

[3.13] gh-130052: Fix search_map_for_section() error handling (GH-132594) #132598

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some error paths do not set Exceptions in _testexternalinspection #130052

Some error paths do not set Exceptions in _testexternalinspection #130052

sergey-miryanov commented Feb 12, 2025 •

edited by bedevere-app bot

Loading

befeleme commented Apr 10, 2025

sergey-miryanov commented Apr 10, 2025

vstinner commented Apr 15, 2025

vstinner commented Apr 15, 2025

befeleme commented Apr 16, 2025

vstinner commented Apr 16, 2025 •

edited

Loading

Some error paths do not set Exceptions in _testexternalinspection #130052

Some error paths do not set Exceptions in _testexternalinspection #130052

Comments

sergey-miryanov commented Feb 12, 2025 • edited by bedevere-app bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

befeleme commented Apr 10, 2025

sergey-miryanov commented Apr 10, 2025

vstinner commented Apr 15, 2025

vstinner commented Apr 15, 2025

befeleme commented Apr 16, 2025

vstinner commented Apr 16, 2025 • edited Loading

sergey-miryanov commented Feb 12, 2025 •

edited by bedevere-app bot

Loading

vstinner commented Apr 16, 2025 •

edited

Loading