coll: add coll_group to collective interfaces #7103
Commits on Sep 14, 2024
-
comm: store num_local and num_external in MPIR_Comm
Store num_local and num_external in MPIR_Comm. Along with internode_table, they help construct internode subgroups.
-
It does not take many instructions to calculate pof2 on the fly. Use of hard coded pof2 prevents collective algorithms to be used for non-trivial coll_group.
-
Lightweight struct to describe sub-groups of a communicator. They intend to replace the subcomms. Preset a set of reserved subgroups to simplify common usages such as intranode group and crossnode group. Since we only expect limited number of dynamic subgroups and they should always be push/pop'ed within the scope, we don't need many dynamic slots.
-
coll: add macros to get rank/size with coll_group
Group collectives will have non-trivial coll_group that alter the rank and size of the communicator. Thease macros and functions will facilitate it.
-
coll: add coll_group argument to coll interfaces
Add coll_group, index to comm->subgroups[], to all collectives except neighborhood collectives.
-
-
-
-
ch4: fallback to mpir if coll_group is non-zero
Assuming the device layer collectives are not able to handle non-trivial coll_group, always fallback when coll_group != MPIR_SUBGROUP_NONE, for now. Also normalize the code style to use the fallback label. We should always fallback to mpir impl routines rather than the netmod routines (composition_beta). The composition_beta may fallback in the future when netmod coll become fancy, resulting in deadloop.
-
coll: threadcomm coll to use MPIR_SUBGROUP_THREADCOMM
Use coll_group=MPIR_SUBGROUP_THREADCOMM for threadcomm collectives. This allows compositional collectives under threadcomm.
-
coll: check coll_group in MPIR_Comm_is_parent_comm
We call MPIR_Comm_is_parent_comm to prevent recursively entering compositional algorithms such as the _smp algorithms. Check coll_group as well as we will switch to use subgroup rather than subcomms. Also check num_external directly for trivial comm. Subcomms and comm->hierarchy_kind will be removed in the future.
-
coll: make non-compositional algorithm coll_group aware
Use MPIR_COLL_RANK_SIZE if the algorithm is topology neutral. Use MPIR_COLL_RANK_SIZE_NO_GROUP if the algorithm is topology dependent. It adds an assertion on coll_group == MPIR_SUBGROUPS_NONE since coll_group may alter the topology assumptions. Intercomm does not work with non-zero coll_group.
-
coll: modify bcast_intra_smp to use subgroups
Replace the usage of subcomms with subgroups.
-
coll: avoid extra intra bcast in bcast_intra_smp
When root is not local rank 0, instead of adding a extra intra-node send/recv or bcast, construct an inter group that includes the root process.
-
-
mpir: replace subcomm usage with subgroups
Directly use information from MPIR_Process rather than from nodecomm in MPIR_Process. One step toward removing subcomms.
-
-
coll: refactor caching tree in the comm struct
Use a single "cached_tree" rather than 3 different fields for each tree type.
-
coll: add coll_group to treealgo routines
The topology-aware tree utilities need check coll_group for correct world ranks.
-
coll: add nogroup restriction to certain algorithms
Some algorithm, e.g. Allgather recexch, caches comm size-related info in communicator, thus won't work with none trivial coll_group. Add a restriction so it will fallback when coll_group != MPIR_SUBGROUP_NONE.
-
coll: check coll_group in MPIR_Sched_next_tag
All subgroup collectives should use the same tag within the parent collectives. This is because all processes in the communicator has to agree on the tag to use, but group collectives may not involve all processes. It is okay to use the same tag as long as the group collectives are always issued in order. This is the case since all group collectives are spawned under a parent collective, which has to obey the non-overlapping rule.
-
coll: refactor barrier_intra_k_dissemination
Because the compiler can't figure out the arithmetic, it is warning: ‘MPIC_Waitall’ accessing 8 bytes in a region of size 0 [-Wstringop-overflow=] Refactor to suppress warning and for better readability. -
coll/allreduce: remove a leftover empty branch
Commit ba1b4dd left an empty branch that should be removed.
