-
Notifications
You must be signed in to change notification settings - Fork 861
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
coll_tuned_dynamic_rules_filename option no way to set alltoall_algorithm_max_requests from the rules file #12589
Labels
Comments
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Sep 14, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Sep 26, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Oct 26, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Nov 1, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Nov 7, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Nov 19, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com> (cherry picked from commit 33f8c74)
burlen
pushed a commit
to burlen/ompi
that referenced
this issue
Nov 19, 2024
Teach the dynamic rules file reader to look for the alltoall_algorithm_max_requests tuning parameter. To keep the dynamic rules file format backward compatible the alltoall_algorithm_max_requests is optional. When not present in the rule definition the value of the corresponding MCA variable is used instead. Resolves open-mpi#12589 Signed-off-by: Burlen Loring <bloring@nvidia.com> (cherry picked from commit 33f8c74)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background information
When using coll_tuned_dynamic_rules_filename option to tune for a specific system there is no way to set the
alltoall_algorithm_max_requests
control variable for thelinear_sync
alltoall algorithm. It's always 0. The rules file parser does not support reading it from the file, and it is not initialized from the mca control variable.What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)
v5.0.3
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
git clone
If you are building/installing from a git clone, please copy-n-paste the output from
git submodule status
.Please describe the system on which you are running
Details of the problem
When using coll_tuned_dynamic_rules_filename option to tune for a specific system there is no way to set the
alltoall_algorithm_max_requests
control variable for thelinear_sync
alltoall algorithm. The rules file parser does not support reading it from the file, and it is not initialized from the mca control variable. As a result it is always 0 when using the rules file.This makes it impossible to reproduce the behavior/performance of the linear_sync algorithm with rules file as compared to when forcing the algorithm via mca control variables, or when using the fixed decision implementations (both of the latter honor the max requests control variable).
Tests using osu_alltoall show that when the max_requests is not set, linear_sync algorithm behaves identically to the linear algorithm. Additionally setting max_requests to 8 on the command line improves alltoall performance over a range of comm and message sizes, but hurts performance over other ranges of comm and message sizes. Therefor, we'd like to be able to set this differently for different comm and message size combos through the dynamic rules file mechanism. At the very least we like to be able to set the max requests control variable on the command line when using the rules file.
The text was updated successfully, but these errors were encountered: