Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Device op cuda #46

Open
wants to merge 76 commits into
base: main
Choose a base branch
from
Open

Device op cuda #46

wants to merge 76 commits into from

Conversation

devreal
Copy link

@devreal devreal commented Nov 6, 2023 •
edited
Loading

Trial PR for feedback from @bosilca. Probably needs some more cleanup but feedback on the design is appreciated. Commits will be squashed later.

Currently only implements offloading for allreduce algorithms. It's missing rooted reduce.

@hppritcha
spack:fix for dlopen missing symbol problem
50731f0 
related to spack/spack#40725

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

1 similar comment
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

@devreal devreal requested a review from bosilca November 6, 2023 00:39
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

@hppritcha
Merge pull request open-mpi#12050 from hppritcha/fix_for_slack_ompi50…
517dcba 
…0_dlopen

spack:fix for dlopen missing symbol problem
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 7, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

2 similar comments
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 7, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 7, 2023

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

  • check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

  • check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

  • check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Joseph Schuchart and others added 21 commits November 7, 2023 18:09
@devreal
Initial draft of CUDA device support for ops
35ff1da 
Signed-off-by: Joseph Schuchart <jschuchart@leconte.icl.utk.edu>
@devreal
First working version of CUDA op support
b7e6f89 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
Update copyright header
164388a 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
Fix minor bugs to get osu_allreduce working
d8110ac 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
cuMemAllocAsync is supported since CUDA 11.2.0
f609127 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
coll/base/allreduce: Condition device allocation on op/dtype support
8ae3dac 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
Make sure the device op callbacks are zero-initialized
655948f 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
Be more graceful when creating a context and stream
7cdc828 
Signed-off-by: Joseph Schuchart <jschuchart@xsdk.icl.utk.edu>
@devreal
fix wrong call to memset
bdb16a1 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Add detector for cudart
5934f43 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Add CUDA stream-based allocator and memory pools
c2c3d0e 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Don't memset the CUDA op component, we need the version
5df449c 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Set the memory pool release threshold
812d068 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Implement device-compatible allocator to cache coll temporaries
a688c84 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Fix devicebucket allocator for larger sizes
bbd362d 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@bosilca @devreal
Fix the RDMA fallback protocol selection.
1fd6636 
If the target process is unable to execute an RDMA operation it
instructs the origin to change the communication protocol. When this
happen theorigin must be informed to cancel all pending RDMA operations,
and release the rdma_frag.

Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
@devreal
Stream-based reduction and ddt copy and 3buff cuda kernels, adopted f…
f2f0f2d 
…or allreduce recursive doubling

Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Remove extra copies from allreduce redscat and ring
8f5b503 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Allow ops and memcpy on managed memory from the host
1c68d17 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
reduce_local: add support for device memory
70dde0f 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Draft of ompi_op_select_device
e603bcc 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
devreal added 28 commits November 7, 2023 18:48
@devreal
Device allocators: correctly handle non-zero ID single accelerator
cfe8a5a 
The accelerator component may report the availability of a single accelerator
whose ID is not zero.

Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
CUDA op: consistently name unsigned_long functions as ulong
3bc7676 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
ROCM op: remove debug output
9c1da7e 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Reduce_local test: correctly test for OPAL_CUDA_SUPPORT and OPAL_ROCM…
a20f671 
…_SUPPORT

These macros are defined to either 1 or 0

Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
More unsigned_long -> ulong fixes in CUDA and ROCm op
97338db 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Fix type in ulong conversion
541b8a0 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Reduce_local: access only host-side memory in error message
8cb2feb 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Make sure CUDA accelerator is initialized before querying number of d…
2996ba0 
…evices

Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Accelerator: provide peak bandwidth estimate
246003f 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
accelerator/rocm: regular memory behaves like unified memory
6601484 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
ROCM: add missing FUNC_FUNC_FN macro
d0fe9a2 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
opal_datatype_accelerator_memcpy: determine device copy type
63b64a0 
We know where source and target buffers are located, so pass the right
transfer direction to the accelerator memcpy call.

Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
accelerator rocm: fix global memcpy stream variable
5a29e13 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Thread base: fix missing include file
5c7c7a1 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Accelerator: Remove debug output
76f00c4 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Allreduce: don't copy inputs if data can be accessed from the host
56bcfee 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Be more careful when releasing temporary receive buffers
a1f089e 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Remove debug output and dead code
33616e6 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Bump max devicebucket allocator max size to 1GB
9da8b54 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
accelerator/cuda: fix error message
93ded5e 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
CUDA: Select compute capability 52 by default
182e6fa 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Sqash const correctness warnings
e5eb45f 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Squash warnings about mismatched function pointer types
14a5372 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Squash printfs
1f63809 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Replace fprintf with show_help
3d9f33a 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Squash compiler warnings
c878c4f 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Clean up cuda and rocm op codes
1c6667d 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Minor tweak to CUDA op configury
7bb4b95 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
@devreal
Fix rebase errors
d1382c3 
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bosilca bosilca Awaiting requested review from bosilca

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@devreal @hppritcha @bosilca