Skip to content

Commit

Permalink
Prevent double UCX initialization in test_dgx (rapidsai#1301)
Browse files Browse the repository at this point in the history
Double initialization of UCX context may raise exceptions and cause test failures, prevent that by reseting the context after doing some initial checks.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: rapidsai#1301
  • Loading branch information
pentschev authored and younseojava committed Apr 16, 2024
1 parent 3fc04db commit 82068de
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions dask_cuda/tests/test_dgx.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,10 @@ def _test_ucx_infiniband_nvlink(
else:
skip_queue.put("ok")

# `ucp.get_active_transports()` call above initializes UCX, we must reset it
# so that Dask doesn't try to initialize it again and raise an exception.
ucp.reset()

if enable_infiniband is None and enable_nvlink is None and enable_rdmacm is None:
enable_tcp_over_ucx = None
cm_tls = ["all"]
Expand Down

0 comments on commit 82068de

Please sign in to comment.