Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coll_tuned_dynamic_rules_filename option no way to set alltoall_algorithm_max_requests from the rules file #12589

Closed
burlen opened this issue May 30, 2024 · 0 comments · Fixed by #12827

Comments

@burlen
Copy link

burlen commented May 30, 2024

Background information

When using coll_tuned_dynamic_rules_filename option to tune for a specific system there is no way to set the
alltoall_algorithm_max_requests control variable for the linear_sync alltoall algorithm. It's always 0. The rules file parser does not support reading it from the file, and it is not initialized from the mca control variable.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

  • Operating system/version: Ubuntu 22.04
  • Computer hardware: Intel Xeon
  • Network type: Infiniband

Details of the problem

When using coll_tuned_dynamic_rules_filename option to tune for a specific system there is no way to set the
alltoall_algorithm_max_requests control variable for the linear_sync alltoall algorithm. The rules file parser does not support reading it from the file, and it is not initialized from the mca control variable. As a result it is always 0 when using the rules file.

This makes it impossible to reproduce the behavior/performance of the linear_sync algorithm with rules file as compared to when forcing the algorithm via mca control variables, or when using the fixed decision implementations (both of the latter honor the max requests control variable).

Tests using osu_alltoall show that when the max_requests is not set, linear_sync algorithm behaves identically to the linear algorithm. Additionally setting max_requests to 8 on the command line improves alltoall performance over a range of comm and message sizes, but hurts performance over other ranges of comm and message sizes. Therefor, we'd like to be able to set this differently for different comm and message size combos through the dynamic rules file mechanism. At the very least we like to be able to set the max requests control variable on the command line when using the rules file.

burlen pushed a commit to burlen/ompi that referenced this issue Sep 14, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen pushed a commit to burlen/ompi that referenced this issue Sep 26, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen pushed a commit to burlen/ompi that referenced this issue Oct 26, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen pushed a commit to burlen/ompi that referenced this issue Nov 1, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
burlen pushed a commit to burlen/ompi that referenced this issue Nov 7, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
@bosilca bosilca closed this as completed in 33f8c74 Nov 7, 2024
burlen pushed a commit to burlen/ompi that referenced this issue Nov 19, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
(cherry picked from commit 33f8c74)
burlen pushed a commit to burlen/ompi that referenced this issue Nov 19, 2024
Teach the dynamic rules file reader to look for the
alltoall_algorithm_max_requests tuning parameter.  To keep the dynamic rules
file format backward compatible the alltoall_algorithm_max_requests is
optional. When not present in the rule definition the value of the
corresponding MCA variable is used instead.

Resolves open-mpi#12589

Signed-off-by: Burlen Loring <bloring@nvidia.com>
(cherry picked from commit 33f8c74)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants