Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wakeup may hang for 20 seconds #15

Open
pentschev opened this issue Apr 3, 2023 · 1 comment
Open

Wakeup may hang for 20 seconds #15

pentschev opened this issue Apr 3, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@pentschev
Copy link
Member

The wakeup mechanism in UCX may hang up to 20 seconds when using the progress thread, and thus appear that there was a deadlock in the application, for example when running:

python python/ucxx/benchmarks/send_recv.py -s 0 -d 1 -o cupy -n 1000000000 -l ucxx-async

There are currently two ways to prevent the hang:

  1. Specify --progress-mode thread-polling to the benchmark above (or UCXPY_PROGRESS_MODE=thread-polling for other applications); or
  2. Specifying a low keep alive interval: UCX_TCP_KEEPINTVL=1ms UCX_KEEPALIVE_INTERVAL=1ms.

Ideally we wouldn't need either one of them. Using thread-polling mode means the progress thread will run at 100% CPU all the time, whereas a low keep alive interval may be the source of other issues, perhaps even causing premature timeouts.
@pentschev pentschev added the bug Something isn't working label Apr 3, 2023
This was referenced Apr 8, 2023
Open
Merged
pentschev added a commit to pentschev/ucxx that referenced this issue Apr 27, 2023
@pentschev
Temporarily increase Python benchmark timeout
0c1f101 
Increase the timeout of the Python benchmarks with `thread` progress
mode to avoid hitting rapidsai#15 . This
can be reverted once that issue is resolved.
@pentschev pentschev mentioned this issue Apr 27, 2023
Merged
@pentschev
Copy link
Member Author

pentschev commented Apr 27, 2023 •
edited
Loading

This also doesn't happen with UCX 1.14+ and protov2 enabled (UCX_PROTO_ENABLE=y), protov2 is expected to become default in UCX 1.16.

pentschev added a commit to pentschev/ucxx that referenced this issue Jun 14, 2023
@pentschev
Set UCX_TCP_KEEPINTVL for Python benchmarks in CI
44f6d68 
Workaround for rapidsai#15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pentschev