MPI_Cart_create_weighted / Topology aware Cartesian communicators

[cniethammer]

Problem

Many scientific applications perform computations on a Cartesian grid. The common approach for the parallelization of these applications with MPI is domain decomposition. To help developers with the mapping of MPI processes to subdomains, the MPI standard provides the concept of process topologies. However, the current interface shows problems and requires too much care in its usage: So, MPI_Dims_create does neither take into account the application nor the hardware topology and most implementations of MPI_Cart_create can only partially fix these problems [Gropp]. However, the mapping has a huge influence on communication performance on multi node NUMA systems.

Proposal

A set of two new API functions is proposed, which takes into account the computational grid as well as the hardware topology.
These functions are MPI_CART_CREATE_WEIGHTED, and MPI_DIMS_CREATE_WEIGHTED.

The new API allows the efficient mapping of processes for applications performing computations on a Cartesian grid over multiple NUMA layers.
For details see the material from the Euro MPI 2018 and 2019 contribution of Niethammer and Rabenseifner: The abstract [Niethammer], slides and implementation example can be found under https://fs.hlrs.de/projects/par/mpi/EuroMPI2018-Cartesian/

The software documentation of the example implementation together with a benchmark see https://fs.hlrs.de/projects/par/mpi//EuroMPI2018-Cartesian/EuroMPI2018_Niethammer+Rabenseifner_ML-Cartesian_software-docu.pdf

The new functions support equally and not equally balanced factorizations of the number of processes of comm_old into n dimensions. Moreover, MPI_CART_CREATE_WEIGHTED provides a reordering of the ranks and allows further optimizations applied to the factorization, taking into account both, application information through the dim_weights argument and hardware information through the comm_old input argument.

Changes to the Text

The addition of the following two API functions to the Process Topology chapter:
MPI_CART_CREATE_WEIGHTED and
MPI_DIMS_CREATE_WEIGHTED,
Together with new section "Cartesian Examples" that explains two use cases for the new routines.

Latest annotated pdf:

Versions for the reading on Feb. 2020 in Portland:
A with MPI_Count dim_weights:

• mpi-report-issue120-topol-2020-02-04-annotated.pdf
  Status: Had reading in Portland

Latest version from Portland WG and plenary discussions:

• mpi-report-issue120-topol-2020-06-14.pdf (without annotation)
• mpi-report-issue120-topol-2020-06-14-annotated.pdf
  Status: Reading of these last changes (Portland-changes in green) is scheduled for the exceptional virtual meeting, June 2020 (and includes also a few last corrections from June 09-14, 2020 in blue)

Impact on Implementations

Implementations have to provide 2 new API functions. No changes to existing MPI functions. Portable Open Source software solution exists.

Impact on Users

Users benefit from improved performance on NUMA systems.

References

[Gropp] W. D. Gropp, Using Node [and Socket] Information to Implement MPI
Cartesian Topologies, Parallel Computing, 2019, and in: Proceedings of the
25th European MPI User' Group Meeting, EuroMPI'18, ACM, New York, NY,
USA, 2018, pp. 18:1-18:9. doi:10.1145/3236367.3236377.
Slides: http://wgropp.cs.illinois.edu/bib/talks/tdata/2018/nodecart-final.pdf

[Niethammer2018] Topology aware Cartesian grid mapping with MPI
Christoph Niethammer and Rolf Rabenseifner, Poster at Euro MPI 2018 https://fs.hlrs.de/projects/par/mpi//EuroMPI2018-Cartesian/EuroMPI2018_Niethammer+Rabenseifner_ML-Cartesian_e-abstract.pdf
and https://fs.hlrs.de/projects/par/mpi/EuroMPI2018-Cartesian/

[Niethammer2019] Christoph Niethammer and Rolf Rabenseifner. An MPI interface for application and
hardware aware Cartesian topology optimization. In Proceedings of the 26th European
MPI Users' Group Meeting (EuroMPI 2019), September 11-13, 2019, Zürich, Switzerland.
ACM, New York, NY, USA, 9 pages.

A set of slides explaining the interface: MPIX_2019-03-Chattanooga_v06.pdf

Introductory slides for the reading at the virtual meetings June 10 and June 29-July 1, 2020:
MPIX_2020-06-10-Munich-v03.pdf

Related Pull Request

https://github.com/mpi-forum/mpi-standard/pull/98

[RolfRabenseifner]

topol-2019-02-17_20.08.pdf uploaded

[cniethammer]

pdf version of the final draft including marked changes:
mpi32-report-topol-2019-02-18-annotated.pdf

[RolfRabenseifner]

pdf version of the final draft including marked changes (additional marks --> version 2):
mpi32-report-topol-2019-02-18-annotated-2.pdf

[RolfRabenseifner]

Slides to explain the new interface / for the reading:
MPIX_2019-03-Chattanooga_v06.pdf

[RolfRabenseifner]

I tried to include most of the results of the discussions at the Chattanooga meeting in March and from the HW topology working group telcons into the following update.
It is also the input for the virtual meeting today (Wed., April 24, 2019):
mpi32-report-topol-2019-04-23-annotated.pdf

[RolfRabenseifner]

I updated the annotations:
mpi32-report-topol-2019-04-23-annotated-2.pdf

[RolfRabenseifner]

I added (nearly) all of the English corrections from Jeff Squyres and all is also available in the pull request:
mpi32-report-topol-2019-04-24-annotated.pdf

It is the base for the virtual meeting today (April 24, 2019)

[RolfRabenseifner]

Latest version for the reading at the WG telcon May 3, 2019:
mpi32-report-topol-2019-05-03-annotated.pdf