Alltoall{,v,w} IN_PLACE fixes

[bwbarrett]

b9012a3 used the alltoallw interpretation of rdisps instead of the
alltoall/alltoallv interpretation. According to the MPI standard,
the byte displacement is recvbuf + rdispls[i] * extent(recvtype) for
alltoall and alltoallv, but is recvbuf + rdispls[i] for alltoallw.

The tag used for alltoallv_in_place was also accidently changed to
TAG_ALLTOALLW in the same commit, so change that back to
TAG_ALLTOALLV.

Signed-off-by: Brian Barrett bbarrett@amazon.com

[bwbarrett]

This should fix #9501, allowing us to put #9329 to bed.

[bosilca]

My bad, I must have switched to something else in the middle of the patch, because I correctly use the extent for the Alltoall, but not for alltoallv. Nice catch.

I went back to my commit and realized that I used the wrong tag for the Alltoall. Would you mind making that change as well ?

[bwbarrett]

I went back to my commit and realized that I used the wrong tag for the Alltoall. Would you mind making that change as well ?

Doh! I should have checked that. Will update.

[bwbarrett]

Moving to draft. This fixes the IBM alltoallv_in_place test, but alltoallv_somezeros now hangs. So this needs a bit more work.

[bosilca]

I found the issue. We are issuing 2 receives per iteration (depending on the counts for the 2 neighbors), and we need to wait for both of them to complete before moving to the next iteration. Thus, if 0 == rcounts[right] but 0 != rcounts[left] we will skip the last wait in the loop. There are two outcomes possible:

1. We might overwrite the send buffer of the next neighbors leading to incorrect result in some corner cases (basically when the left at one iteration becomes the right at the next).
2. We might deadlock because some requests are not waited on, so MPI_Finalize might block.

[bwbarrett]

@bosilca can you re-review after the cleanups?