Skip to content
This repository has been archived by the owner on Dec 21, 2023. It is now read-only.

Scenarios test failure and core dump from image_classifier test #3079

Closed
guihao-liang opened this issue Mar 31, 2020 · 5 comments
Closed

Scenarios test failure and core dump from image_classifier test #3079

guihao-liang opened this issue Mar 31, 2020 · 5 comments
Labels
bug p1

Comments

@guihao-liang
Copy link
Collaborator

FYI, this error log is from scenario_tests_linux_python36. This also happens to scenario_tests_linux_python27 but without this verbose error log.

toolkits/test_image_classifier.py::ImageClassifierSqueezeNetTest::test_classify double free or corruption (out)
Fatal Python error: Aborted

Thread 0x00007fd831ffb700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd8327fc700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd832ffd700 (most recent call first):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd8337fe700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108 in worker
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd833fff700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108 in worker
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd85cff9700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108 in worker
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd85d7fa700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108 in worker
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fd9704c5740 (most recent call first):
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/turicreate/data_structures/sframe.py", line 2917 in random_split
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/turicreate/toolkits/_supervised_learning.py", line 324 in create
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/turicreate/toolkits/classifier/logistic_classifier.py", line 325 in create
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/turicreate/toolkits/image_classifier/image_classifier.py", line 296 in create
  File "/build/scenario-tests/tests/toolkits/test_image_classifier.py", line 100 in setUpClass
  File "/build/scenario-tests/tests/toolkits/test_image_classifier.py", line 264 in setUpClass
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/unittest.py", line 99 in fixture
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 788 in call_fixture_func
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 964 in pytest_fixture_setup
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 914 in execute
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 584 in _compute_fixture_value
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 503 in _get_active_fixturedef
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 487 in getfixturevalue
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/fixtures.py", line 477 in _fillfixtures
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/unittest.py", line 120 in setup
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 373 in prepare
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 123 in pytest_runtest_setup
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 244 in from_call
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 94 in runtestprotocol
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/main.py", line 247 in _main
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/build/scenario-tests/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/build/scenario-tests/venv/bin/pytest", line 8 in <module>
Aborted (core dumped)
@TobyRoseman
Copy link
Collaborator

@guihao-liang - I can not replicate this on macOS 10.15.4 with Python 3.6. Is this a Linux only issue? Is the issue intermittent or does it happen consistently?

@guihao-liang
Copy link
Collaborator Author

it's from Linux: scenario_tests_linux_python36. Yes, consistent. There's a segfault across almost of our Linux tests. I had a glance at it and it involved TensorFlow.

@TobyRoseman
Copy link
Collaborator

What version of TensorFlow is getting used?

We don't work with TensorFlow 2.1 (the current version) on Linux, see #3003.

@guihao-liang
Copy link
Collaborator Author

What version of TensorFlow is getting used?

We don't work with TensorFlow 2.1 (the current version) on Linux, see #3003.

Yeah, whatever TF required by the current 6.1.

@guihao-liang
Copy link
Collaborator Author

It's fixed by #3089

@TobyRoseman TobyRoseman mentioned this issue Apr 9, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug p1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TobyRoseman @guihao-liang