Running tests fails with "node down: Not properly terminated", maybe execnet related?

[jenyacazacu]

We recently moved to Python 3.4 and when running our tests with pytest-xdist installed and in parallel with (-n) sometimes the tests fail with the below error. The traceback is from using faulthandler. The tests consistently fail during tests that use celery tasks and that have the format:

@override_settings(task_always_eager=True)
def test_example(self):

Packages used:
execnet-1.5.0
pytest-3.0.6
xdist-1.15.0

Here is the error log:

platform linux -- Python 3.4.5, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
Django settings: gameserver.settings.test (from environment variable)
rootdir: /home/jcazacu/Repos/main_repo/ares-game-server, inifile: tox.ini
plugins: xdist-1.15.0, faulthandler-1.4.1, django-3.1.2, cov-2.5.1, celery-4.0.2
gw0 [535] / gw1 [535] / gw2 [535] / gw3 [535]
scheduling tests via LoadScheduling
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Fatal Python error: Segmentation fault

Thread 0x00007fbd16d3d700 (most recent call first):
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 386 in read
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 418 in from_io
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Current thread 0x00007fbd1eed2740 (most recent call first):
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 335 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/utils.py", line 62 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 288 in _savepoint_rollback
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 328 in savepoint_rollback
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/transaction.py", line 243 in exit
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1004 in _rollback_atomics
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1066 in _fixture_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 908 in _post_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/channels/tests/base.py", line 57 in _post_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 216 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/unittest.py", line 157 in runtest
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 104 in pytest_runtest_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 163 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 133 in call_and_report
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 79 in runtestprotocol
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "", line 77 in run_tests
File "", line 61 in pytest_runtestloop
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 133 in _main
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 98 in wrap_session
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 127 in pytest_cmdline_main
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "", line 159 in
File "", line 1 in do_exec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1072 in executetask
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 259 in integrate_as_primary_thread
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1047 in serve
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1534 in serve
File "", line 8 in
File "", line 1 in
......[gw1] node down: Not properly terminated
fReplacing crashed slave gw1

Any ideas?

Currently we just moved the task tests to run serially since the failures are so unpredictable.

[RonnyPfannschmidt]

its really not clear whats happening there, its just clear that python itself gets seg-faulted, which is quite a feat and a real pain to debug

[jenyacazacu]

ok, thx, do you have any suggestions as to where I can start with debugging?

[RonnyPfannschmidt]

unfortunately not, im not familiar with the surrounding libs in use

[kydbakep]

[gw4] [ 96%] PASSED tests/warehouse/test_warehouse_supplier_refunds.py::TestDeleteSupplierRefund::test_delete_refund_after_delete_cell
10:32:56
[gw5] [ 96%] PASSED tests/warehouse/test_warehouse_supplier_refunds.py::TestDeleteSupplierRefund::test_delete_refund_after_good_has_become_serial
10:32:56
[gw3] node down: Not properly terminated
10:32:56
[gw3] [ 96%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_close_residue_rule_via_mask
10:32:56

10:32:56
replacing crashed worker gw3
10:32:56

10:32:56
[gw8] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9
10:32:57

10:32:57
[gw8] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0]
10:33:00

10:33:10
tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_positive
10:33:10
[gw2] node down: Not properly terminated
10:33:10
[gw2] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[Abc]
10:33:10

10:33:10
replacing crashed worker gw2
10:33:10

10:33:10
[gw9] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9
10:33:10

10:33:10
[gw9] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0]
10:33:13

10:33:16
tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[ ]
10:33:16
[gw7] node down: Not properly terminated
10:33:16
[gw7] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestCreateAndDeleteResidueControl::test_check_table_icon
10:33:16

10:33:16
replacing crashed worker gw7
10:33:16

10:33:16
[gw10] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9
10:33:17

10:33:17
[gw10] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0]
10:33:20

10:34:01
tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[0]
10:34:01
[gw0] node down: Not properly terminated
10:34:01
[gw0] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_close_residue_rule_modal
10:34:01

10:34:01
replacing crashed worker gw0
10:34:01

10:34:01
[gw11] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9
10:34:02

10:34:02
[gw11] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0]

[espears1]

I have seen this type of error occur when combining pytest-xdist with other libraries that deal with multithreaded or multi-processing code. One common pain point is the Annoy library from Spotify, especially older versions. It does many multi-threaded actions without end user control of the threading, and there are just extremely mystrerious and un-debuggable ways that pytest-xdist causes some bad interaction with threading to lead to a segfault and a crashed worker. Worse this can be different for different OSes or Python versions.

I am debugging an example right now where on Mac OS with Python 3.6.9, everything is fine and pytest-xdist runs my test suite (with parallel workers) without issue. If I take the same code and put it in an equivalent Ubuntu Docker image and run the tests, I get a segfault from execnet and crashed workers. But if I run the tests serially in the Docker image, all tests pass.

These issues are virtually impossible to boil down to simplified reproducible examples as well, since if I knew what all the factors were that are required to minimally reproduce it, that would probably be the solution to debugging it. It's very, very hard to use pytest-xdist in these situations.

[brechtm]

I'm seeing a similar error (Python segfault + "node down: Not properly terminated"). It seems to fail consistently in my Ubuntu CI runs (these logs will be unavailable soon) and on my MacBook, but only on CPython 3.11 (3.11.0-beta.3 on Ubuntu and 3.11.0b3+ [f9d0240] on macOS).

nox > python run_tests.py -m 'not longrunning' --script-launch-mode=subprocess -s tests_regression

============================= test session starts ==============================
platform linux -- Python 3.11.0b3, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/runner/work/rinohtype/rinohtype, configfile: setup.cfg
plugins: assume-2.4.3, github-actions-annotate-failures-0.1.6, console-scripts-1.3.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
gw0 I / gw1 I
gw0 [88] / gw1 [88]
.................../home/runner/work/rinohtype/rinohtype/tests_regression/rst/columns_two.rst:1: (ERROR/3) Undefined substitution referenced: "problematic".
....../home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:11: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
/home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:18: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
/home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:21: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
............../home/runner/work/rinohtype/rinohtype/tests_regression/rst/inline_markup.rst:1: (ERROR/3) Undefined substitution referenced: "problematic".
.................Fatal Python error: Segmentation fault
Thread 0x00007f92f6d6f700 (most recent call first):
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 400 in read
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 432 in from_io
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
Current thread 0x00007f92f7854740 (most recent call first):
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 466 in <genexpr>
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 466 in _index
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 473 in get_rowspanned_columns
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 562 in __int__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 575 in __iter__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 537 in __eq__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 303 in match
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 361 in match
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 622 in match
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 739 in find_matches
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 742 in find_matches
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 742 in find_matches
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 327 in get_matches
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 697 in _get_value_lookup
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/attribute.py", line 469 in get_value_for
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/attribute.py", line 333 in get_config_value
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 512 in get_style
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/util.py", line 160 in function_wrapper
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/draw.py", line 86 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 386 in draw_cell_border
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 401 in _place_rows_and_render_borders
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 166 in render_rows
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 176 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 581 in _flow_with_next
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 637 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 366 in _render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 306 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 197 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 197 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 178 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/template.py", line 461 in render
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 478 in _render_pages
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 436 in render
  File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 105 in render_doctree
  File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 92 in _render_file
  File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 59 in render_rst_file
  File "/home/runner/work/rinohtype/rinohtype/tests_regression/test_rst.py", line 45 in test_rst
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/python.py", line 1761 in runtest
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 110 in run_one_test
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 91 in pytest_runtestloop
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 291 in <module>
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1084 in executetask
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1060 in serve
  File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
Extension modules: markupsafe._speedups (total: 1)
...........[gw1] node down: Not properly terminated
F
replacing crashed worker gw1
s...................
=================================== FAILURES ===================================
_________________________ tests_regression/test_rst.py _________________________
[gw1] linux -- Python 3.11.0 /home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/bin/python
worker 'gw1' crashed while running 'tests_regression/test_rst.py::test_rst[png]'
=============================== warnings summary ===============================
.nox/regression-3-11-sdist/lib/python3.11/site-packages/babel/messages/catalog.py:13
.nox/regression-3-11-sdist/lib/python3.11/site-packages/babel/messages/catalog.py:13
...

I have seen this type of error occur when combining pytest-xdist with other libraries that deal with multithreaded or multi-processing code.

All tests perform image comparison in parallel (diffpdf.py), but the crash does not occur in this stage of the test but rather during rinohtype's rendering which is single-threaded.

It's always test_rst[png] that segfaults. When running without pytest-xdist (-n 0), all is well. It is possible to reproduce the issue running only the test_rst[png] test case:

git clone https://github.com/brechtm/rinohtype.git
cd rinohtype
git checkout bd4b4157
poetry install
poetry run nox -r -s "regression-3.11(wheel)" -- -k png

• pytest (7.1.2)
• pytest-xdist (2.5.0)

UPDATE: Running the tests not through nox, the issue doesn't occur!

.nox/regression-3-11-wheel/bin/python run_tests.py -m 'not longrunning' --script-launch-mode=subprocess -k png tests_regression

[WenjieDu]

Confronted with the same problem in PyPOTS testing here https://github.com/WenjieDu/PyPOTS/actions/runs/4577128474/jobs/8082161160

[basnijholt]

No solution, but for others that every come across this issue, in my case the following happened.

When running pytest tests in parallel with xdist, my tests crash with:

[gw5] node down: Not properly terminated
[gw5] FAILED tests/test.py::test_with_botorch

replacing crashed worker gw5

This crash happens only when I have:

class A:
    def x(self):
        import botorch # only used in this function

but not when

import botorch

class A:
    def x(self):

Also cannot reproduce it on MacOS but only in the CI on DevOps with ubuntu-latest.

[gonchik]

@basnijholt
Just curious, do you still detecting the same construction?
did you played with number of threads etc?