Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix issue with unmatched messages in ph5diff #3719

Merged
merged 1 commit into from
Oct 19, 2023
Merged

Fix issue with unmatched messages in ph5diff #3719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions release_docs/RELEASE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -791,6 +791,19 @@ Bug Fixes since HDF5-1.14.0 release

Tools
-----
- Fixed an issue with unmatched MPI messages in ph5diff

The "manager" MPI rank in ph5diff was unintentionally sending "program end"
messages to its workers twice, leading to an error from MPICH similar to the
following:

Abort(810645519) on node 1 (rank 1 in comm 0): Fatal error in internal_Finalize: Other MPI error, error stack:
internal_Finalize(50)...........: MPI_Finalize failed
MPII_Finalize(394)..............:
MPIR_Comm_delete_internal(1224).: Communicator (handle=44000000) being freed has 1 unmatched message(s)
MPIR_Comm_release_always(1250)..:
MPIR_finalize_builtin_comms(154):

- Fixed an issue in h5repack for variable-length typed datasets

When repacking datasets into a new file, h5repack tries to determine whether
Expand Down
3 changes: 0 additions & 3 deletions tools/lib/h5diff.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1485,9 +1485,6 @@ diff_match(hid_t file1_id, const char *grp1, trav_info_t *info1, hid_t file2_id,
} /* end else */
} /* end while */

for (i = 1; (int)i < g_nTasks; i++)
MPI_Send(NULL, 0, MPI_BYTE, (int)i, MPI_TAG_END, MPI_COMM_WORLD);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This same loop already exists in phdiff_dismiss_workers in tools/lib/h5diff.c and gets called by the manager rank on ph5diff exit. This loop was just causing the same messages to be sent twice and end up unmatched.


/* Print any final data waiting in our queue */
print_incoming_data();
} /* end if */
Expand Down
8 changes: 4 additions & 4 deletions tools/src/h5diff/ph5diff_main.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ ph5diff_worker(int nID)
char filenames[2][MAX_FILENAME];

/* Retrieve filenames */
MPI_Recv(filenames, MAX_FILENAME * 2, MPI_CHAR, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &Status);
MPI_Recv(filenames, MAX_FILENAME * 2, MPI_CHAR, 0, MPI_TAG_PARALLEL, MPI_COMM_WORLD, &Status);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Match MPI_TAG_PARALLEL from above rather than just any message tag


/* disable error reporting */
H5E_BEGIN_TRY
Expand Down Expand Up @@ -173,7 +173,7 @@ ph5diff_worker(int nID)

/* When get token, send all of our output to the manager task and then return the token */
for (i = 0; i < outBuffOffset; i += PRINT_DATA_MAX_SIZE)
MPI_Send(outBuff + i, PRINT_DATA_MAX_SIZE, MPI_BYTE, 0, MPI_TAG_PRINT_DATA,
MPI_Send(outBuff + i, PRINT_DATA_MAX_SIZE, MPI_CHAR, 0, MPI_TAG_PRINT_DATA,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manager rank specifies MPI_CHAR when receiving these messages. It doesn't seem to make much of a difference anyway for char vs. byte, but it seems better to make sure we match the types anyway.

MPI_COMM_WORLD);

/* An overflow file exists, so we send it's output to the manager too and then delete it */
Expand All @@ -188,15 +188,15 @@ ph5diff_worker(int nID)
while ((tmp = getc(overflow_file)) >= 0) {
*(out_data + i++) = (char)tmp;
if (i == PRINT_DATA_MAX_SIZE) {
MPI_Send(out_data, PRINT_DATA_MAX_SIZE, MPI_BYTE, 0, MPI_TAG_PRINT_DATA,
MPI_Send(out_data, PRINT_DATA_MAX_SIZE, MPI_CHAR, 0, MPI_TAG_PRINT_DATA,
MPI_COMM_WORLD);
i = 0;
memset(out_data, 0, PRINT_DATA_MAX_SIZE);
}
}

if (i > 0)
MPI_Send(out_data, PRINT_DATA_MAX_SIZE, MPI_BYTE, 0, MPI_TAG_PRINT_DATA,
MPI_Send(out_data, PRINT_DATA_MAX_SIZE, MPI_CHAR, 0, MPI_TAG_PRINT_DATA,
MPI_COMM_WORLD);

fclose(overflow_file);
Expand Down