OpenMPI: MPI_IN_PLACE malfunctioning

[MarcelHB]

I'm evaluating R&R MPI wrappers for an HPC application. This simulation makes use of MPI_IN_PLACE buffer value calls such as MPI_Allgather or MPI_Allreduce, resulting in various kinds of errors when using ReMPI.

Please consider the following minimal sample, that is simply supposed to sum all ranks everywhere:

PROGRAM sample_allreduce
  USE mpi
  IMPLICIT NONE

  INTEGER :: ierr
  INTEGER :: rank, rank_in_place
  INTEGER :: rank_sum

  CALL MPI_Init(ierr)
  CALL MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

  rank_in_place = rank

  PRINT *, 'Rank: ', rank
  CALL MPI_Allreduce(rank, rank_sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD, ierr)
  CALL MPI_Allreduce(MPI_IN_PLACE, rank_in_place, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD, ierr)
  PRINT *, 'Sum: ', rank_sum, ' - ', rank_in_place

  CALL MPI_Finalize(ierr)
END PROGRAM sample_allreduce

Let's say I launch this sample via OpenMPI v2.0.2, Linux v4.11 x86-64 and GNU Fortran Compiler v6.3.0, and mpiexec -np 3, I receive the following and correct result:

 Rank:            0
 Rank:            1
 Rank:            2
 Sum:            3  -            3
 Sum:            3  -            3
 Sum:            3  -            3

Now, I try the same with REMPI_MODE=0 and rev. 21ae26b of ReMPI, this results in bad values (0 instead of 3) for the MPI_IN_PLACE call:

 Rank:            1
 Rank:            2
 Rank:            0
 Sum:            3  -            0
 Sum:            3  -            0
 Sum:            3  -            0

In addition, there is some corruption when being used in the HPC Fortran application, resulting in this error after calling MPI_Allgather and MPI_IN_PLACE for sendbuf argument:

*** An error occurred in MPI_Allgather
*** reported by process [287899649,3]
*** on communicator MPI COMMUNICATOR 11 SPLIT FROM 5
*** MPI_ERR_TYPE: invalid datatype

This is what made me investigating further. For now, I cannot post any of its code, and I hope the sample gives enough reproducibility for this issue.

According to this statement: nerscadmin/IPM#6 (comment), in case of MPI_IN_PLACE (and MPI_BOTTOM), there is no implementation-agnostic way of relaying a Fortran mpi_allgather_ (or any similar method) symboled call of a wrapper properly to the C domain PMPI_Allgather as in ReMPI. Passing values through to Fortran pmpi_allgather_ instead is one way to handle this.

[jprotze]

There is code to handle the translation of MPI_IN_PLACE, it is just commented out:
https://github.com/PRUNERS/ReMPI/blob/master/src/rempi_pmpi.cpp#L193

Can you try and remove the comment in line 193 and comment out line 194? I think, this should fix your issue.

[MarcelHB]

Yes, I already tried this. I any case, the programs die very early by segmentation fault, such as (-O2):

Thread 1 "allreduce_sample" received signal SIGSEGV, Segmentation fault.
0x00007fda9d30b74f in MPI_Comm_rank_fortran_wrapper (comm=<optimized out>, ierr=0x7ffe09147fec, rank=0x7ffe09147fe8) at rempi_pmpi.cpp:1557
1557	  _wrap_py_return_val = MPI_Comm_rank(MPI_Comm_f2c(*comm), (int*)BufferC2F((int*)rank));
(gdb) bt
#0  0x00007fda9d30b74f in MPI_Comm_rank_fortran_wrapper (comm=<optimized out>, ierr=0x7ffe09147fec, rank=0x7ffe09147fe8) at rempi_pmpi.cpp:1557
#1  mpi_comm_rank__ (comm=<optimized out>, rank=0x7ffe09147fe8, ierr=0x7ffe09147fec) at rempi_pmpi.cpp:1575
#2  0x000056549508ad3b in sample_allreduce () at allreduce_sample.f90:15
#3  0x000056549508af31 in main (argc=1, argv=0x7ffe0914a0a3) at allreduce_sample.f90:2
#4  0x00007fda9c9b63f1 in __libc_start_main (main=0x56549508aefb <main>, argc=1, argv=0x7ffe091480f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe091480e8) at ../csu/libc-start.c:291
#5  0x000056549508abea in _start ()

[kento]

Hi @MarcelHB ,

Thank you for reporting this issue. Let me take a look.
If you find a way to fix this, please let me know or just send me the pull request.

Thanks !
Kento

[MarcelHB]

Hi @kento

I think I got something. If, for my setup, I define BufferC2F (better naming it BufferF2C?) like this:

#define BufferC2F(x) ((x) == (void*) &mpi_fortran_bottom_ ? MPI_BOTTOM : ((x) == (void*) &mpi_fortran_in_place_ ? MPI_IN_PLACE : (x)))

my example works. In gdb, of all the tests in the macro right now, these symbols were the only ones available, positioned correctly for the in-place/bottom flags. So I guess there are two things to do:

1. Re-enable mapping BufferC2F by default, or at least allow this by some -D flag.
2. Restructuring of these expressions to actually work.

I cannot really help there as I have no clue of older MPI-implementations, and have nothing but OpenMPI at hand right now to test this.

Thank you!

[kento]

Hi @MarcelHB ,

Thank you for the workaround idea.
I'll incorporate it.

Thanks !
Kento

[dongahn]

@kento and @MarcelHB, do we want to add some simple fortran test coverage into our test suite?

Perhaps we should also consider adding some travis CI interation.

I was talking with LANL and AWE users yestersay and they asked a lot about fortran. At least we have a real good fortran path to Archer with flang and xlflang supporting threadsanitizer, and probably rempi would be a matter of hardening...

[MarcelHB]

I cannot participate in those measures for now, but I'd suggest to really have that case above tested for Fortran. From my observation so far on MPI-wrapping and Fortran code, this is one of the more interesting cases, for being weaker in the interop. specs.

[jprotze]

@kento and @dongahn: I think, instead of trying to pick the right symbol as in the weak symbol approach, a cleaner solution would be to call into fortran and get the value from there. We started to do this in wrap to get MPI_STATUS_SIZE: https://github.com/LLNL/wrap/blob/fortran-fixes/getStatusSize.f

The fortran code is then called from C as in https://github.com/LLNL/wrap/blob/fortran-fixes/wrap.py#L136 .

Especially on machines with different frontent/backend architectures, there is no reliable way to identify the right values at configure or build time.

[kento]

Thank you for the advise @jprotze.
Let me fix it.

Is this MPI_STATUS_SIZE issue related to the MPI_IN_PLACE issue ? Or, are they independent problems each ?

[jprotze]

I pushed my suggestion to the new branch. I have no idea of how to include the fortran source file into the Makefile.
Two options of using the code:

• use MPI_F_IN_PLACE and MPI_F_BOTTOM in the BufferC2F:
  #define BufferC2F(x) (MPI_F_BOTTOM == x ? MPI_BOTTOM : (MPI_F_IN_PLACE == x ? MPI_IN_PLACE : (x)))
• only call the fortran functions once and store the value in a static variable.

And finally, the fortran part needs to be optional and depends on the availability of a fortran compiler.

[kento]

Hi @jprotze,

Thank you for the suggestion. I had time to take a look at this issue today.

I looked at your code and I have a question. In my environment (MVAPICH2/2.2), MPI_IN_PLACE and MPI_BOTTOM (of mpif.h) are both zero. This means the current BufferC2F cannot even distinguish between MPI_BOTTOM and MPI_IN_PLACE. Then, both MPI_BOTTOM and MPI_IN_PLACE (of mpif.h) become MPI_BOTTOM (of mpi.h) under the current BufferC2F.

Therefore, assuming x is MPI_BOTTOM or MPI_IN_PLACE,

#define BufferC2F(x) (MPI_F_BOTTOM == x ? MPI_BOTTOM : (MPI_F_IN_PLACE == x ? MPI_IN_PLACE : (x))) 

is actually

#define BufferC2F(x) (0 == 0 ? MPI_BOTTOM : (0 == 0 ? MPI_IN_PLACE : (x))) 

However, in mpi.h, MPI_BOTTOM is 0, and MPI_IN_PLACE is -1. So we cannot call MPI_Allreduce with MPI_IN_PLACE via the PMPI wapper.

I think we need to implement BufferC2F depending on OpenMPI, MPICH, MVAPICH and others.
How do you think ? If I'm misunderstanding something, please let me know.

Thanks,
Kento

[kento]

Hi @MarcelHB ,

Saying that, MPI_Allreduce and many other MPI functions do not need to be wrapped unless --enable-cdc is used. I fixed it to wrap only when CDC is enabled.
So you will not see the MPI_IN_PLACE reduction issue in the latest version.

Thank you for reporting this issue. If you see any other issues, please let me know

Thanks !
Kento

[MarcelHB]

Will try this, thanks for your help!

[MarcelHB]

Hello again,

Does your commit/the repo lack this rempi_fortran.f90 file? Right now, a freshly generated Makefile fails in make complaining about its absence.

Marcel