[WIP] A new shared memory collectives component #10470
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a copy of coll/sm leveraging the opal/smsc component for single-copy collectives. Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Children wait for all their segments to be consumed, parents wait at the start of an op for their children to complete. Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
|
@devreal What's the status of this? I assume this is post v5.0 feature? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds
coll/smdirect, a clone ofcoll/smthat relies on cross-process memory mapping as provided by XPMEM. In contrast tocoll/sm, data is not copied into an intermediate buffer but buffers are registered with xpmem and the access keys exchanged. Processes can then copy data directly from the source to the target buffer. A similar synchronization mechanism using atomic flags is used to wait for data availability. We currently implemented broadcast, barrier, reduce, and allreduce (as a combination of reduce and bcast). Eventually, this component should replace the (apparently unmaintained)coll/smcomponent.Below are some performance measurements on Hawk (2x64 core AMD EPYC system installed at HLRS), min/max/avg taken from OSU benchmarks: the new component shows good bandwidth for larger messages. For small messages, however, both
coll/tunedandcoll/smshow significantly lower minimum times (and thus lower average) due to buffering (intermediate buffer in coll/sm, eager messages in coll/tuned). This will be addressed incoll/smdirectin future work by buffering small messages in a pre-registered buffer. Interesting are the maximum times (the longest time any process spends in the collective), wherecoll/smdirectis competitive for reduce operations and provides significant improvements for large broadcasts. The current implementation ofcoll/allreduceIn the OSU benchmarks, the barrier implementation in
coll/smis faster (2.5us) thancoll/smdirect(5.3us) sincecoll/smis set up to partially overlap the execution of two consecutive barriers. Both are faster thancoll/tuned(6.3us) though.This PR is work in progress. Things to do:
coll/sm, to improve latency.coll/han(for the node-local portion).This PR includes the fixes to
smsc/xpmemin #10127 and needs to be rebased once that is merged.