Fix MPI Data plane cohort handling

[vicentebolea]

We have been observing scalability issues with the MPI DP. I have been able to narrow down to potential root causes and I was able to propose possible solutions to it in this PR.

• Task id collision between different ranks at scale. This was expected to happens since we used the process id (PID) as the rank id (not a good idea). This has been replaced by an aggregate ID of the process PID and the machine hostid.

• Currently in the event of a reader rank failure the MPI Dataplane write simply frees and disconnect to every resource pertaining to its cohort. Since at large scale the failure of readers nodes (or timeout event) are not that rare, in this PR I make the Write to skip releasing resources pertaining to the reader unless all the peers of the cohorts have exited. To make happy MPI, I do perform a MPI_Port_disconnect since otherwise the "failing" node might get stuck (in case of a non critical error).

Possibly fixes: #3456
Possibly fixes: #3439
Possibly fixes: #3378

@franzpoeschel I was able to run the given reproducer in Crusher without getting any error with this changes.

[eisenhauer]

I assume not quite ready yet since you still have printf's in it, but generally looks good.

[franzpoeschel]

Hey Vicente, this is great to hear!
I understand correctly that this is already in a usable state so I can test this out in the next days, using our actual simulation code?

[vicentebolea]

Hey Vicente, this is great to hear! I understand correctly that this is already in a usable state so I can test this out in the next days, using our actual simulation code?

Yes, please, if you can test this out that will be great!

[suchyta1]

How scalable is this expected to be? Or SST in general?

I have a test where I start from using and XGC restart file, to be sent to a reader. Each writing process reads one block of data from the BP file. By block, I mean an ADIOS block, where one block is from one of the original XGC writer process' send dispatch. To mimic the way XGC runs, there are 8 senders per node, and for now, I was just doing one reader, but I can turn that up too. The run worked at 64 sender nodes, and took about 7 seconds a single reader to receive the data. (I get around needing multiple readers by using MPMD.) But when I double to 128 sender nodes, the reader hangs. That's about 176 GB of data over 1024 blocks. Increasing the number of readers doesn't prevent the hang. I've tried adjusting some of the engine parameters too, but they don't seem to have impact relative to hanging.

[eisenhauer]

How scalable is this expected to be? Or SST in general?

In general, the expectation is "highly scalable". As always the answer in practice depends upon a lot of things, what is scalable on one machine may not be on another, resource availability can be an issue at all levels, etc. A hang may mean a bug like a race condition, it may mean that an operation failed silently leaving SST waiting for a result that'll never come, or pretty much anything else. If possible, the first step is to turn on verbosity to figure out what is hanging. I can't speak to what level of verbosity is in the MPI data plane, but setting the environment variable "SstVerbose" to a numerical value between 1 and 5 is a place to start. 5 is the highest verbosity.

[suchyta1]

I looked through the code a little, and did a rerun with CMVerbose=1 and CMLowLevelVerbose=1, so the log attached now has more information.

slurm-1432345.out.log.zip

[eisenhauer]

I looked through the code a little, and did a rerun with CMVerbose=1 and CMLowLevelVerbose=1, so the log attached now has more information.

Thanks. Those are important for the WAN transport and for the control plane. Probably don't play a role in whatever problem is happening in the MPI data plane, but giving Vicente more output doesn't hurt.

[franzpoeschel]

I reverted the workarounds that I used so far and ran an example on Frontier at 128 nodes.
With the rolled-back workarounds, there is now one instance of N:1 and one instance of 1:M communication. Seems to work without trouble with this :)

EDIT: I tried if I can actually reproduce the original issue when using the ADIOS2 master branch instead of this. It seems that I can't, will need to have a closer look. Maybe my setup was not weird enough to trigger the issue.

[vicentebolea]

@suchyta1 thanks for reporting issues with the MPI DP. I Will take a look to it. Quick question: are you using the tip commit of this PR MR?

@franzpoeschel I am happy that this fix your reproducer app.

[suchyta1]

Yes, that's what I was using.

[suchyta1]

I added some extra debugs, and as far as I can tell, there's a lock operation that is never returned from.
slurm-1432539.out.zip

[eisenhauer]

Yes, that tracks. The interesting bit here is the bit from the reader-side data plane on rank 0 (which is the "DP Reader 0" line).

DP Reader 0 (0x1004380): Waiting for completion of memory read to rank 652, condition 3855,timestep=0, is_local=0
Writev success
DP Reader 0 (0x1004380): Remote read; CMCondition_wait
Before unlockCManager Unlock at "/ccs/home/esuchyta/software/build/frontier/adios2-gcc12.2.0-test/thirdparty/EVPath/EVPath/cm_interface.c" line 632
After unlockBefore lockCManager Lock at "/ccs/home/esuchyta/software/build/frontier/adios2-gcc12.2.0-test/thirdparty/EVPath/EVPath/cm_interface.c" line 84

From the first line above, we know we're in the MPI data plane, in MpiWaitForCompletion(), and it has fallen into the CMCondition_wait() at line 571. That we never fall out of this means that the corresponding CMCondition_signal (at line 708 at the end of the MPIReadReplyHandler() subroutine) never happens. So, what's going on? I don't know, but it's certainly something at the dataplane level. Just what isn't clear and it's probably going to require the attention of the author @vicentebolea .

[vicentebolea]

@eisenhauer it appears that its waiting for the rank=652 (writer) to reply its message however the rank=652 has long been either dead or unresponsive.

Few questions I have noticed that this rank has the same ID has the rank=748, is this odd? See below:

DP Writer 652 (0x102c3a0): MpiReadRequestHandler: Replying reader=0 with MPI port name=tag#0$connentry#0140AE23$
DP Writer 748 (0x102c3a0): MpiReadRequestHandler: Replying reader=0 with MPI port name=tag#0$connentry#01408A24$

Similarly it seems that this setup of a single reader and 800 writers might cause to have 800 MPI connections on the reader side simultaneously, maybe hitting some limit of the mpi client/server implementations. I am working on a reproducer to further debug this. I will also investigate how to add such as test in the CI.

[eisenhauer]

Few questions I have noticed that this rank has the same ID has the rank=748, is this odd? See below:

The ID printed after the rank is the base address of the writer-side stream structure. If an application has more than one output stream open, this differentiates them in the output. But in your case it just means that two ranks happened to allocate the same base address for the stream structure. They're different processes, so this shouldn't matter. (They'd probably be the same more often if most OS' didn't use memory randomization as a security measure.)

Anyhow, no help in this situation I'm afraid...

[franzpoeschel]

Using this, I now had a full-scale run on Frontier from 9126(nodes) x 8(GPUs) writers to 9126(nodes) readers. Data exchange happened only node-internally, 6Gb were exchanged per step and node (not much, but I was focusing on scaling the application to a large number of writers/readers for now).
I have not (yet) done a full evaluation of performance, but by squinting at the numbers, I/O time looks stable.
The workaround described in #3846 was necessary to scale above 7000 nodes, but that has nothing to do with this PR.

[vicentebolea]

@eisenhauer do you think that we should have this merged before waiting the workaround of @franzpoeschel to be included (in a more generic bugfix form) in master?

[eisenhauer]

@eisenhauer do you think that we should have this merged before waiting the workaround of @franzpoeschel to be included (in a more generic bugfix form) in master?

I would go ahead and merge. @franzpoeschel workaround is an orthogonal issue that just happens to also come up when we scale up.