trying to identify timing out tests on Mac

[ReimarBauer]

To get to this information I first created a set of all test names, pytest --collect-only parsed the content for the functions name. Then I logged the pytest -n auto --dist loadfile of executed tests until it gets stuck.

I did several runs, always waiting until it stucks, then I compared both sets.

• 4 {'test_window_start', 'test_switch_view', 'test_kwargs_update_does_not_harm', 'test_window_contact_none'}
• 15 {'test_identify_CF_isopressure', 'test_num2date', 'test_identify_CF_hybrid', 'test_jsec_to_datetime', 'test_datetime_to_jsec', 'test_kwargs_update_does_not_harm', 'test_identify_CF_isopottemp', 'test_identify_CF_isopotvort', 'test_parse_iso_duration', 'test_parse_iso_datetime', 'test_identify_CF_coordhybrid', 'test_identify_CF_time', 'test_identify_CF_isoaltitude', 'test_get_latlon_data', 'test_identify_CF_ensemble'}
• 23 {'test_get_operation_by_id', 'test_get_operations_skip_archived', 'test_get_users_with_permission', 'test_uploads', 'test_messages', 'test_set_last_used', 'test_authorized', 'test_hello', 'test_register_user', 'test_home', 'test_message_attachment', 'test_undo_changes', 'test_dont_create_operation', 'test_changes', 'test_get_operations', 'test_get_users_without_permission', 'test_get_auth_token', 'test_version_name_delete', 'test_check_login', 'test_unauthorized_profile_image_upload', 'test_set_active', 'test_kwargs_update_does_not_harm', 'test_user_register_handler'}
• 4 {'test_window_start', 'test_switch_view', 'test_kwargs_update_does_not_harm', 'test_window_contact_none'}
• 4 {'test_kwargs_update_does_not_harm', 'test_window_start', 'test_window_contact_none', 'test_switch_view'}
• 4 {'test_window_contact_none', 'test_kwargs_update_does_not_harm', 'test_window_start', 'test_switch_view'}
• 35 {'test_xml', 'test_xml_time_multiperiod', 'test_invalid_schema', 'test_server_service_cache', 'test_xml_inittime_reference', 'test_server_abort_getmap', 'test_no_schema', 'test_invalid_url', 'test_xml_time_forecast', 'test_filter_handling', 'test_multilayer_handling', 'test_xml_time_error', 'test_xml_emptyextent', 'test_xml_othertimeformat', 'test_window_start', 'test_server_getmap_cached', 'test_forward_backward_clicks', 'test_xml_time_init_period', 'test_xml_separate_leafs', 'test_kwargs_update_does_not_harm', 'test_switch_view', 'test_singlelayer_handling', 'test_xml_currenttag', 'test_valid_before_init', 'test_xml_time_period', 'test_multilayer_syncing', 'test_xml_no_inittime', 'test_xml_no_validtime', 'test_connection_error', 'test_xml_onlytimedim', 'test_xml_no_elevation', 'test_xml_separatedim', 'test_server_no_thread', 'test_multilayer_drawing', 'test_window_contact_none'}

It gives more or less a random result. It is likly not a test the reason for the problems.

keyboard interrupt shows always something

^CTraceback (most recent call last):
  File "/Users/reimarbauer/miniforge3/envs/mssdev/bin/pytest", line 10, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201, in console_main
    code = main()
           ^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/config/__init__.py", line 175, in main
    ret: ExitCode | int = config.hook.pytest_cmdline_main(config=config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py", line 330, in pytest_cmdline_main
    return wrap_session(config, _main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py", line 318, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/logging.py", line 868, in pytest_sessionfinish
    return (yield)
            ^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/terminal.py", line 893, in pytest_sessionfinish
    result = yield
             ^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/warnings.py", line 141, in pytest_sessionfinish
    return (yield)
            ^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/xdist/dsession.py", line 99, in pytest_sessionfinish
    nm.teardown_nodes()
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/xdist/workermanage.py", line 108, in teardown_nodes
    self.group.terminate(self.EXIT_TIMEOUT)
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/execnet/multi.py", line 237, in terminate
    safe_terminate(
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/execnet/multi.py", line 348, in safe_terminate
    reply.get()
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/execnet/gateway_base.py", line 327, in get
    self.waitfinish(timeout)
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/execnet/gateway_base.py", line 334, in waitfinish
    if not self._result_ready.wait(timeout):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/threading.py", line 629, in wait
    signaled = self._cond.wait(timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/threading.py", line 327, in wait
    waiter.acquire()
KeyboardInterrupt

 /Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py:318: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 4371712080, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py:318: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 4471325136, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py:318: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 4440489744, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/Users/reimarbauer/miniforge3/envs/mssdev/lib/python3.11/site-packages/_pytest/main.py:318: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 14107635856, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(``

[anj20]

hi @ReimarBauer
I think the failing test cases are due to race conditions and shared state issues when running tests in parallel using
pytest -n auto --dist loadfile.
i think we should add more @pytest.mark tags to better categorise the parrallel test execution

[ReimarBauer]

The problem happens also with serial testing.