Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IntegrationClassification2D timeout #6311

Closed
wyli opened this issue Apr 6, 2023 · 4 comments
Closed

IntegrationClassification2D timeout #6311

wyli opened this issue Apr 6, 2023 · 4 comments
Assignees
@wyli

Comments

@wyli
Copy link
Contributor

wyli commented Apr 6, 2023

Describe the bug

[2023-04-06T04:23:52.950Z] ======================================================================
[2023-04-06T04:23:52.950Z] ERROR: test_timing (__main__.IntegrationClassification2D)
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------
[2023-04-06T04:23:52.950Z] Traceback (most recent call last):
[2023-04-06T04:23:52.950Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/tests/utils.py", line 633, in _wrapper
[2023-04-06T04:23:52.950Z]     raise timeout_error
[2023-04-06T04:23:52.950Z] multiprocessing.context.TimeoutError: 'test_timing' in '__main__' did not finish in 1000s.
[2023-04-06T04:23:52.950Z] 
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------

(this happens randomly on A100 postmerge, works fine v100)
@wyli wyli self-assigned this Apr 6, 2023
@wyli
Copy link
Contributor Author

wyli commented Apr 6, 2023

seems to be addressed by limiting num_threads

export OMP_NUM_THREADS=8
export MKL_NUM_THREADS=8

@wyli wyli closed this as completed Apr 6, 2023
@wyli
Copy link
Contributor Author

wyli commented Apr 11, 2023

Still an issue

@wyli wyli reopened this Apr 11, 2023
@Nic-Ma
Copy link
Contributor

Nic-Ma commented Apr 11, 2023

@KumoLiu is testing on A100.

Thanks.

wyli added a commit that referenced this issue Apr 11, 2023
@wyli
workaround for #6311
8d61255 
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli wyli mentioned this issue Apr 11, 2023
Merged
wyli added a commit that referenced this issue Apr 11, 2023
@monai-bot @mingxin-zheng
@liamchalcroft @wyli @KumoLiu
auto updates (#6324)
9ef42ff 
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

(closing #6318 closing #6319 auto3dseg)
(closing #6314 closing #6315 dtype conversion)
(closing #6326 closing #6329 metatensor clone)
(including a workaround for #6311)

---------

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
Signed-off-by: Mingxin Zheng <18563433+mingxin-zheng@users.noreply.github.com>
Signed-off-by: Liam Chalcroft <liam.chalcroft.20@ucl.ac.uk>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Co-authored-by: Mingxin Zheng <18563433+mingxin-zheng@users.noreply.github.com>
Co-authored-by: Liam Chalcroft <liam.chalcroft.20@ucl.ac.uk>
Co-authored-by: Wenqi Li <wenqil@nvidia.com>
Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com>
@wyli wyli closed this as completed Apr 11, 2023
@wyli wyli reopened this Apr 11, 2023
@wyli
Copy link
Contributor Author

wyli commented Aug 10, 2023 •
edited
Loading

no issue after the blossom migration

@wyli wyli closed this as completed Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wyli wyli

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wyli @Nic-Ma