Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macos testing flaky #24

Open
klieret opened this issue Oct 16, 2023 · 1 comment
Open

Macos testing flaky #24

klieret opened this issue Oct 16, 2023 · 1 comment
Labels
CI/CD help wanted Extra attention is needed

Comments

@klieret
Copy link
Member

klieret commented Oct 16, 2023

Every now and then, testing on macos fails with

============================= test session starts ==============================
platform darwin -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /Users/runner/work/object_condensation/object_condensation
configfile: pyproject.toml
testpaths: tests
plugins: cov-4.1.0
collected 5 items

tests/test_losses_tf.py .                                                [ 20%]
Fatal Python error: Aborted

Thread 0x00007000109f4000 (most recent call first):
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/concurrent/futures/thread.py", line 78 in _worker
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 870 in run
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x000070000f9f1000 (most recent call first):
<no Python frame>

Thread 0x000070000f868000 (most recent call first):
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/selectors.py", line 415 in select
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/multiprocessing/connection.py", line 931 in wait
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/concurrent/futures/process.py", line 362 in _queue_management_worker
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 870 in run
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x000000010cff0600 (most recent call first):
  File "/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/torchinductor_runner/dc/cdc4kf4ezbrshemad7yt3qlunt3c6hvs3cad5pysycsdhtyjkhkm.py", line 84 in call
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 401 in _run_from_cache
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 374 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1594 in rng_functionalization_wrapper
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1506 in call_func_with_args
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2533 in runtime_wrapper
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1482 in g
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 3905 in forward
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_dynamo/external_utils.py", line 17 in inner
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 328 in _fn
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/object_condensation/pytorch/losses.py", line 111 in <resume in _condensation_loss_tiger>
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/object_condensation/pytorch/losses.py", line 111 in <resume in _condensation_loss_tiger>
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/object_condensation/pytorch/losses.py", line 111 in _condensation_loss_tiger
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 328 in _fn
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/object_condensation/pytorch/losses.py", line 204 in condensation_loss_tiger
  File "/Users/runner/work/object_condensation/object_condensation/tests/test_losses_torch.py", line 73 in test_condensation_loss_tiger
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python.py", line 1792 in runtest
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 325 in _main
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pytest/__main__.py", line 5 in <module>
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 87 in _run_code
  File "/Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 194 in _run_module_as_main
/Users/runner/work/_temp/e097f47f-389b-45b5-b3b8-57a027848c68.sh: line 1:  9781 Abort trap: 6           python -m pytest -ra --cov --cov-report=xml --cov-report=term --durations=20
tests/test_losses_torch.py .
Error: Process completed with exit code 134.

problem can be fixed by re-running failed tests

@klieret
Copy link
Member Author

klieret commented Oct 16, 2023

Might be kevlened/pytest-parallel#93, but this should be fixed since py3.6

klieret added a commit that referenced this issue Oct 16, 2023
@klieret klieret changed the title Macos testing fails for py3.8 Macos testing fails Oct 16, 2023
@klieret klieret changed the title Macos testing fails Macos testing flakey Oct 16, 2023
@klieret klieret changed the title Macos testing flakey Macos testing flaky Oct 16, 2023
@klieret klieret added CI/CD help wanted Extra attention is needed labels Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD help wanted Extra attention is needed
Projects
None yet
1 participant