libnbc nonuniform type failures in ibcast, iallgather(v)

[jjhursey]

When testing coll/libnbc with nonuniform data types three of the algorithms failed. I'll post a link to the test case in the comments. The unit test only works with -np 5 some of the tests pass fine with -np 4.

• ibcast Falls into an infinite error loop in libnbc
• iallgather and iallgatherv produce wrong answers. Might be the same underlying problem.

ibcast Failure

Note: This test will enter an infinite loop displaying MPI Error in MPI_Testall() (18) until PR #2245 is resolved.

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 0 
 0 /  5) Running MPI_Ibcast...
 1 /  5) Running MPI_Ibcast...
 2 /  5) Running MPI_Ibcast...
 3 /  5) Running MPI_Ibcast...
 4 /  5) Running MPI_Ibcast...
MPI Error in MPI_Testall() (18)
MPI Error in MPI_Testall() (18)
...

iallgather Failure

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 1
 0 /  5) Running MPI_Iallgather...
 1 /  5) Running MPI_Iallgather...
 2 /  5) Running MPI_Iallgather...
 3 /  5) Running MPI_Iallgather...
 4 /  5) Running MPI_Iallgather...
 3 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 3 /  5) Failed in chkbuf. err = -1
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -2.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 2 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 2 /  5) Failed in chkbuf. err = -6
 1 /  5) buf[2] : actual 2 [2], expected 1 [1]. from 0
 1 /  5) Failed in chkbuf. err = -2
 4 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 4 /  5) Failed in chkbuf. err = -6

iallgatherv Failure

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 2
 0 /  5) Running MPI_Iallgatherv...
 1 /  5) Running MPI_Iallgatherv...
 2 /  5) Running MPI_Iallgatherv...
 3 /  5) Running MPI_Iallgatherv...
 4 /  5) Running MPI_Iallgatherv...
 0 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 0 /  5) Failed in chkbuf. err = -6
 3 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 3 /  5) Failed in chkbuf. err = -6
 4 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 4 /  5) Failed in chkbuf. err = -6
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -2.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 2 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 2 /  5) Failed in chkbuf. err = -6

[jjhursey]

Here is the test case for this issue:
https://gist.github.com/jjhursey/dfbb38c7c82dcde7d0e05e62852fce3c

Sorry it's not more general, but should help reproduce the problem.

[ggouaillardet]

@jjhursey i fixed MPI_Iallgather[v] in #2245

the issue with MPI_Ibcast is the very same and unresolved issue we have in coll/tuned.
we should not try to split a message into chunks since we cannot be sure the remote peer is able to split the message into chunks of the same size.
for example, if you need to MPI_Send 1M MPI_INT, it is trivial to MPI_Send 1k MPI_INT 1024 times (and this is what we do in ibcast and coll/tuned
but if the remote peer MPI_Recv one vector of 1M MPI_INT, there is currently no way to split that into 1024 MPI_Recv of partial vector.
we currently assume we do not run into this case and the remote peer will be just fine, and once in a while, we are reminded the assumption is wrong.
fwiw, i started to work on #1789 as a proof of concept, but had not much time to finalize it

[jjhursey]

@ggouaillardet I can confirm that your commit fixed the problem with the MPI_Iallgather and MPI_Iallgatherv collectives. Thanks!

Adding a reference to Issue #1763 - which describes the data corruption problem with MPI_Bcast. Since those two are related problems, a fix for one should be ported to the other.

As a workaround I might look into adding an MCA parameter to skip the algorithm selection based upon message size for bcast.

[ggouaillardet]

@jjhursey IIRC, coll/tuned has a (per collective ?) segment_size MCA param, and we recommend it is set to 0 when running into this kind of pattern. You might want to use the same naming scheme for libnbc.
An other workaround is to blacklist coll/tuned, but that cannot be mimicked for non-blocking collectives.

[ggouaillardet]

Btw, if coll/tuned are used, the default behavior is MPI_ERR_TRUNCATE causes the program abort.
If coll/libnbc is used instead, should we follow the same logic (e.g. call the error handler, which aborts by default) ?

[jjhursey]

I created PR #2330 which adds a new MCA parameter to avoid this datatype size based decision. It's not ideal (since the NBC_BCAST_CHAIN algorithm will not be selected), but should work around the problem until it can be fixed.

Let me know if you have any ideas for improvements on that PR.

[hppritcha]

maybe this should be moved to 2.1 for a more complete solution?

[jjhursey]

Yeah. I'll PR over my bcast parameter to the 2.(0.)x series. Then we can move it to v2.1 for a complete bcast solution.

[jjhursey]

We talked about this on the call today. Once the two work around PRs are merged (#2378, #2379). Then we will close this issue as fixed.

Once that happens then we should followup with a fix on Issue #1763 (I've added a note there with a reference)

[jjhursey]

I'm closing this issue now that the work around has been put into the v2.(0.).x branches. Followup work will be on Issue #1763