UCX: Hang combining exclusive/shared window lock

[devreal]

Running Open MPI 4.0.1 in combination with Open UCX 1.5 I am seeing my application hang while one process attempts to release an exclusive lock while the target attempts to acquire a shared lock. The code below can be used to reproduce the issue (tested on our IB cluster):

#include <stdio.h>
#include <mpi.h>

int main(int argc, char **argv)
{
  MPI_Win win;
  int elem_per_unit = 1;
  int *baseptr;
  int rank, size;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  MPI_Win_allocate(
    elem_per_unit*sizeof(int), 1, MPI_INFO_NULL,
    MPI_COMM_WORLD, &baseptr, &win);

  if (size == 2) {
    // get exclusive lock
    if (rank != 0) {
      int val;
      printf("[%d] Acquiring exclusive lock\n", rank);
      MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
      MPI_Put(&val, 1, MPI_INT, 0, 0, 1, MPI_INT, win);
      MPI_Win_flush(0, win);
    }

    MPI_Barrier(MPI_COMM_WORLD);

    // release exclusive lock
    if (rank != 0) {
      printf("[%d] Releasing exclusive lock\n", rank);
      // Rank 1 hangs here
      MPI_Win_unlock(0, win);
    }
  }

  // Rank 0 hangs here
  printf("[%d] Acquiring shared lock\n", rank);
  MPI_Win_lock_all(0, win);

  MPI_Win_unlock_all(win);
  MPI_Win_free(&win);
  MPI_Finalize();

  return 0;
}

Build with:

$ mpicc mpi_shared_excl_lock.c -o mpi_shared_excl_lock

Run with:

$ mpirun -n 2 -N 1 ./mpi_shared_excl_lock
[1] Acquiring exclusive lock
[1] Releasing exclusive lock
[0] Acquiring shared lock

Interestingly, leaving out the barrier between acquiring and releasing the lock lets the example run successfully. Also, things run fine when using Open IB instead of UCX.

[yosefe]

@devreal what is OpenMPI configure command and git revision?

[devreal]

I used Open MPI version 4.0.1 downloaded from the website and configured using:

./configure CC=icc CXX=icpc FTN=ifort --with-ucx=/path/to/ucx-1.5.0/ --without-verbs --enable-debug

It's the Intel compiler in version 19.0.1. I could try changing to the GNU compiler but I'm not sure that that makes a difference, let me know if I should give it a shot.

[gpaulsen]

@jladd-mlnx Any progress?

[jladd-mlnx]

@janjust please take it.

[janjust]

@devreal I have a hard time reproducing this issue, what is the ucx and ompi commit?

[devreal]

@janjust That was done using UCX 1.5.0 and Open MPI 4.0.1, both built from release branches. I will give the 4.0.x branch a try with the latest UCX release and report back.

[devreal]

@janjust I cannot reproduce this on latest 4.0.x (git 368da00) with UCX 1.6.x (git 0309365) so I'm closing this issue. Thanks for checking though :)

[janjust]

@devreal , thanks - I believe a lot of your issues are sw atomic related