MPI_Comm_idup hang in v2.0.x with MPI_THREAD_MULTIPLE

[jjhursey]

I created a test case in ompi-tests underibm/communicator/comm_idup_many.c

This test is doing a simple:

    for (i = 0; i < num_comms_to_create; i++) {
        MPI_Comm_idup(MPI_COMM_WORLD, &comms[i], &req[i]);
    }

    MPI_Waitall(num_comms_to_create, req, MPI_STATUSES_IGNORE);

If you compile the v2.0.x branch with --enable-mpi-thread-multiple this test will hang when you try to create more than one communicator this way.

./configure --enable-debug --enable-mpi-thread-multiple

The problem is that the first request will progress past stage one, but the second request will block the progress thread in this section of code:

• https://github.com/open-mpi/ompi/blob/v2.0.x/ompi/communicator/comm_cid.c#L517-L530

The master and v2.x branches pass this test. So I believe that the CID allocation fix that went into those two branches 'fixed' this problem.

[jjhursey]

As part of Issue #2234 it was requested that we port a portion of this change to the v2.0.x series to fix a spawn issue:

• COMM_SPAWN broken in v2.0.x #2234 (comment)

I believe that this commit to the v2.x branch is what we would be looking at:

• v2.x communicator code updates #2215

[jsquyres]

Per the call today, #2381 may fix this issue.

[jsquyres]

Confirmed by @jjhursey: fixed by #2381.