Deadlock in H5FD__mpio_open due to bad MPI_File_get_size

[adammoody]

While debugging another issue unrelated to HDF5, I think I tripped up a deadlock condition. In my test, it seems the MPI_File_get_size call returns an error here:

hdf5/src/H5FDmpio.c

Lines 828 to 832 in c56464f

	/* Only processor p0 will get the filesize and broadcast it. */
	if (mpi_rank == 0) {
	if (MPI_SUCCESS != (mpi_code = MPI_File_get_size(fh, &size)))
	HMPI_GOTO_ERROR(NULL, "MPI_File_get_size failed", mpi_code)
	} /* end if */

That causes rank 0 to jump to the bottom of the function to close the file, which is a collective:

hdf5/src/H5FDmpio.c

Lines 858 to 861 in c56464f

	done:
	if (ret_value == NULL) {
	if (file_opened)
	MPI_File_close(&fh);

While other ranks are stuck in a different collective here:

hdf5/src/H5FDmpio.c

Line 835 in c56464f

if (MPI_SUCCESS != (mpi_code = MPI_Bcast(&size, (int)sizeof(MPI_Offset), MPI_BYTE, 0, comm)))

Also, I noticed that there may be other patterns like this while grepping the file to look for this location, like maybe:

hdf5/src/H5FDmpio.c

Lines 1223 to 1229 in c56464f

	/* Read on rank 0 Bcast to other ranks */
	if (file->mpi_rank == 0)
	if (MPI_SUCCESS !=
	(mpi_code = MPI_File_read_at(file->f, mpi_off, buf, size_i, buf_type, &mpi_stat)))
	HMPI_GOTO_ERROR(FAIL, "MPI_File_read_at failed", mpi_code)
	if (MPI_SUCCESS != (mpi_code = MPI_Bcast(buf, size_i, buf_type, 0, file->comm)))
	HMPI_GOTO_ERROR(FAIL, "MPI_Bcast failed", mpi_code)